Crescent

[molten wyvern]

Crescent is my fifth attempt at making an x86-64 kernel, though I have tried to made it be as platform-independent as possible so it should be easy to port it to a new platform in the future. Currently it has virtual memory allocation using vmem, tsc/apic timers using hpet for calibration, io-apic for interrupts, ps2 keyboard support, smp support, multilevel feedback queue scheduler, and a tty that I am currently working on.

As for the design I am not completely sure yet, but I am pretty sure its going to be a mix of traditional unixy-things and some custom approaches (sometimes inspired by windows).

I feel like the scheduler might need a bit more explaining, so Ill try to explain it here. It has an adjustable amount of levels where each lower level has lower timeslice than the one after it (the min and max timeslices are adjustable). The scheduler always runs tasks in a round-robin fashion starting from the topmost level going down if there are no more tasks. The tasks initially begin at the topmost level which has the shortest timeslice and if they keep on running for the whole slice then they are being put on the level below and that continues until it either reaches the deepest level or it blocks eg. waiting for something. After it wakes up again after being blocked its level is going to be bumped one level up so short tasks that block often are preferred over long-running tasks. In my implementation I also added a manually chosen task-priority which currently is only used when deciding whether to pre-empt the currently running task when waking a task up or not, but I plan on using it in some other places too.

And finally here is the github link 😛 https://github.com/Qwinci/crescent

[fallen sparrow]

looks great

[molten wyvern]

I finally got around to implementing some basic interrupt priority stuff and also a better IRQ system (it supports sharing interrupts too)

next I am likely going to change the way the scheduler works in quite a significant way adding yet one more layer of levels to it so I can truly have priorities between tasks on the same time slice level. so it would be that the old levels for the slice microseconds each get levels inside them for the different priorities (or ranges maybe) and then those contain the tasks

[molten wyvern]

I decided to try to make some sort of "desktop" and its kinda working™️ except that the cursor redrawing is pretty visible if you move it

(I also accidentally recorded desktop audio into it but meh, don't watch with audio if you don't want to listen to some random music)

[stable hollow]

cool! a breath of fresh air to see someone not just port xorg or wayland or whatever, good luck with that!

[molten wyvern]

ty

Ill have to also implement some kind of ipc or whatever because rn this is all inside one process

(after I first make the windows look a bit more like proper windows, add some controls and child windows)

[unreal flame]

nice work

[molten wyvern]

I managed to somewhat implement nested windows (though I also broke something about the dirty regions and the cursor also feels more laggy when moving so idk if I actually draw unnecessary stuff or smth now)

[molten wyvern]

and it doesn't pagefault anymore if you move a "window" below the bottom edge of the screen (it would just basically freeze the entire thing with no output because I don't re-enable fb logging even if the process that mapped the fb died)

[molten wyvern]

I fixed most of the bugs except one where if you move the cursor over the top bar of a window then the parts of child windows that are at that same x position get erased until you move something directly over them or drag the parent window

[molten wyvern]

apparently that bug was caused by a missing bounds check in the drawing code

[molten wyvern]

buttons

they are kinda weird tho because if you move the mouse outside of the button after having clicked it it stays in the clicked state until you hover over it at which point it actually runs the callback and returns to the normal state

I guess Ill have to implement some kind of mouse leave events to make them be less weird

(and the reason why it dies at the middle is that I made it do a shutdown when clicked and the other button does a reboot but in qemu it doesn't really work)

[molten wyvern]

I did that, it was just a matter of checking whether the old mouse position was within the window content rect if the new mouse position isn't

so literally just one if more

[stable hollow]

Looks like a perfect place for a screamer

[molten wyvern]

lol

[molten wyvern]

I implemented process spawning, now I guess Ill implement some kind of ipc after first thinking how exactly do I want to do it

[molten wyvern]

I think I am going to do ipc using sockets at least to start with

its likely not going to be the most efficient thing ever but it should work good enough

and as a bonus then I have a generic socket api so I can hook up my udp stuff to it (along with tcp but I haven't yet implemented that)

[molten wyvern]

I got the api and service advertisement implemented but I didn't yet implement the actual ipc socket because I hit an annoying bug that looks like something overwrites one byte past allocated memory so the next free node's previous link address ends up as 0xff80017ffffb90cb

actually no its the last byte of that qword that's zeroed for some reason and not the first

[dire galleon]

nice progress so far

debugging issues like that is not fun lol

[molten wyvern]

there should be some kind of way to attach sizes to pointers then it would automatically update the size when you modify the pointer and check it when you dereference to make sure that it is within the allowed range

I guess that could also be implemented as a class but it would be a huge pain to refactor almost all pointers to use it

I also tried to implement asan but the problem is that it needs memory allocations and a page map that is currently in use that I can modify so I have to do allocator init and whatever before running constructors which would mean marking all those functions as no_sanitize_address and doing memory init before running the constructors ends up initializing some globals and then when I run the constructors they get zeroed

[molten wyvern]

I think I found the bug, I changed my shared_ptr type's constructors around so they work properly with inherited types to be of the form ```cpp
template<typename U = T>
constexpr shared_ptr(const shared_ptr<U>& other) {
ptr = other.ptr;
refs = other.refs;
if (ptr) {
++*refs;
}
}

it started working after I restored the non-templated constructors

anyway I guess now I can actually get to implementing the sockets

[molten wyvern]

it works now I just have to properly disconnect the sockets if either side of it dies as right now it would just access some freed memory and die

it wasn't that bad code-wise too, its only like 500 lines (including the service stuff)

after I have fixed that Ill have to create some kind of protocol you can use to create windows and get access to a framebuffer for that specific window which means that Ill have to also implement shared memory for it to not be super slow

[dire galleon]

nice

[dire galleon]

yeah I've considered the idea for my next rewrite, some kind of 'debug ptr' you can attach metadata to.

and when you dont care for debugging all that extra stuff is removed at compile time

[molten wyvern]

got shared memory done (at least it worked according to my initial testing)

and the current api for it that I came up with is pretty simple ```cpp
int sys_shared_mem_alloc(CrescentHandle& handle, size_t size);
int sys_shared_mem_map(CrescentHandle handle, void** ptr);
int sys_shared_mem_share(CrescentHandle handle, CrescentHandle process_handle, CrescentHandle& result_handle);

basically you first allocate a shared memory object and after that you can map it or create a duplicate handle of it that is accessible in an other process (that the other process can then map or share again)

and well you have to first transmit the handle to the other process by whatever way you want (most likely it would be ipc)

or the other process could just guess the correct handle by trying to feed sequential integers to the shared mem syscalls and seeing if it comes out as invalid argument or not

one thing which I do have to fix at some point is making the kernel shared_ptr type thread-safe, right now its not really an issue because I haven't implemented a load balancer yet and everything is just spawned on the same cpu but when I do then I need to make the refcount updates atomic

[molten wyvern]

anyway I guess tomorrow (well technically today lol) I am going to try implementing some kind of windowing protocol thing and maybe also udp sockets (which shouldn't be that bad)

then I also need file syscalls but thats just a matter of making wrapper syscalls over the vnode functions

[cyan slate]

Do you still work on qacpi?

[molten wyvern]

well I did fix a test regression a few days ago that was caused by the changes I made to reduce stack usage but other than that I haven't yet done anything else (well except implementing bankfields)

I should also implement indexfield reading/writing but for that I'll have to change the way my field reading/writing works

and maybe try doing events (though not sure if thats something that should be within the lib or the kernel)

[molten wyvern]

I did manage to implement something but while doing so I discovered some kind of bug where the current thread in gsbase is the idle thread but then it tries to execute code from an another process and then just pagefaults and kills the entire kernel

or alternatively gp faults after trying to execute code at address zero, thats probably related too

[molten wyvern]

I fixed those, one of them was related to the fact that I queued the idle thread when it can just randomly be switched to but I am not completely sure why would it cause those weird things

and then there were some random userspace pagefaults caused by out of bounds access to the framebuffer so it just corrupted something on the userspace heap

[molten wyvern]

(that window with the red square is created using the windowing protocol and a mapped framebuffer)

I should add some kind of borders tho

also the protocol client api rn ```cpp
windower::Windower windower;
if (auto status = windower::Windower::connect(windower); status != 0) {
puts("[console]: failed to connect to window manager");
return 1;
}
windower::Window window;
if (auto status = windower.create_window(window, 0, 0, 400, 300); status != 0) {
puts("[console]: failed to create window");
return 1;
}

if (window.map_fb() != 0) {
    puts("[console]: failed to map fb");
    return 1;
}
auto* fb = static_cast<uint32_t*>(window.get_fb_mapping());
for (uint32_t y = 0; y < 32; ++y) {
    for (uint32_t x = 0; x < 32; ++x) {
        fb[y * 400 + x] = 0xFF0000;
    }
}

well its an abstraction over the actual protocol that works over ipc but anyway

and currently its kinda limited as you can only create a framebuffer backed window but not even destroy it

[molten wyvern]

now the windows have borders

window closing is implemented now too

also the desktop isn't thread safe, I should make the desktop thread safe before I forget and then wonder why some random pagefaults happen when having windows spawned by multiple threads

actually wait no, it doesn't need to be thread-safe

well some stuff needs to be because there is a different thread listening and accepting the ipc connections but other than that

also I realized that I need some way to get a handle to the process on the connecting side of an ipc socket to be able to share the framebuffer memory to it

I guess some kind of getpeername equivalent would work for that, shouldn't be too hard to implement

[molten wyvern]

I implemented that and also indexfields in qacpi (not sure if they actually work tho but I think they should)

now it dies on ssdt load on my desktop because of an unimplemented match op

[cyan slate]

Match op is complex on paper but in reality nt only supports ints so its easy

[molten wyvern]

that's good to know, so it just converts the package elements to ints and then compares the element with both of the matches and if both match then it returns the index

well I don't think it would be that hard to implement for strings or buffers either but more annoying for sure

[molten wyvern]

now it boots on my desktop too, I am not sure if the shutdown works because I don't really have any good way to trigger it as it doesn't have any kind of ps2 keyboard emulation (though ig I could again call it inside kmain)

[cyan slate]

enable sci and fixed event 2(?)

[molten wyvern]

yeah I should do that too

and the ec memes

though I also lack the api to add/remove a region handler rn

[molten wyvern]

I tried to implement window close buttons but I discovered some userspace malloc bugs in the process and apparently there are even more but I haven't yet managed to trigger them outside of my kernel in linux userspace

[molten wyvern]

tbh I am not sure if its the malloc or the mmap, it may as well be the mmap as even if I make malloc straight up return mmap'd memory giving qemu 32gb of ram it generates a billion page protection violations (because I don't actually kill the process if its properly mapped)

actually it looks like the stack ends up changed to read only somehow and I have no idea why

[ivory cloak]

oberrow hell incoming

[molten wyvern]

yeah lol

these kinds of bugs are also very annoying to debug

[molten wyvern]

it turned out to be a stack overflow because I made a mistake that caused infinite recursion

at least I learned that you should have guard pages and implemented that so I don't have to spend an another day debugging this kind of thing

[molten wyvern]

apparently I got hda working on my desktop (though its hardcoding in the sound path rn, but it isn't that hard to do properly)

I were missing one verb for enabling the headphone amplifier and output on the pin complex

and also I improved the initialization a little so it doesn't screw up the whole codec when used with vfio passthrough (previously it got into a completely unusable state where it needed a host reboot to get it back working because I were resetting the hda without first stopping the dma engines and streams)

after I have implemented the signal path discovery properly I think I might look into what it would take to get it to work on my laptop (as it has intel smartsound technology and afaik that needs some firmware from sof-firmware to work)

[dire galleon]

congrats, thats exciting 😄

[molten wyvern]

it looks like it even works on my laptop using the integrated speakers from what I read in linux source I think the firmware is needed if you want to use other formats than pcm16/pcm32

(and by some luck it happened to work using the exact same hard-coded widget configuration as I had for qemu)

[molten wyvern]

[kernel][x86]: EXCEPTION: general protection fault at 7F9E4830FFFF8001
[kernel][x86]: killing user process console
[kernel][sched]: destroying exited thread console
[kernel][sched]: destroying empty process console

[kernel][x86]: pagefault caused by kernel read from FFFF80017F9E4A30 (instruction fetch)
    at FFFF80017F9E4A30
``` I just discovered some random faults that kill the whole kernel and they only happen once in like 30 qemu launches

[hot cobalt]

hate that sort of bug

[molten wyvern]

yeah these are annoying to debug, from the looks of it it looks like both of them actually happen in kernel mode and are somehow related to interrupts

likely the stack somehow gets messed up and then the iretq causes the cpu to jump to some garbage address

[molten wyvern]

and the backtrace is pretty useless as well

[winged parrot]

reverse debugging?

[molten wyvern]

?

oh yeah, I wonder if it would work

[winged parrot]

or that cool qemu plugin that lets you trace rip values maybe

[winged parrot]

rr my beloved

[molten wyvern]

can you use that with qemu?

[winged parrot]

no, but i think qemu includes its own meme

[molten wyvern]

it gets stuck when trying to replay it for some reason

(well this is not a record with the bug but I just wanted to try it)

[winged parrot]

i see

sucks

[molten wyvern]

actually I got it to work, I just needed to add the cdrom manually and add some replay memes ```
qemu-system-x86_64 -m 4G -M q35,smm=off -smp 1 -no-shutdown -no-reboot -cpu qemu64,+umip,+smep,+smap -serial stdio -boot d -drive file=image.iso,if=none,snapshot,id=img-direct -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay -device virtio-scsi-pci,id=scsi0 -device scsi-cd,bus=scsi0.0,drive=img-blkreplay -icount shift=auto,rr=record,rrfile=replay.bin

now I just need to run qemu in record mode until I happen to capture the bug ig

[winged parrot]

fun

[molten wyvern]

the only problem is that it hasn't happened in a single run over the couple minutes I have been running that command

doesn't look like it is going to happen

[molten wyvern]

I got the bug fixed thanks to JW's tips (#osdev-misc-0 message) so now I can return to actually working on stuff again

[molten wyvern]

I finally implemented legacy pci interrupts, the reason I wanted to avoid that in the first place was that it involved parsing acpi resource descriptors from _CRS but it wasn't that bad after all

the thing that I mostly wanted that for was rtl8139 in qemu because I have rtl8169 with msi on real hw but its nice being able to test networking stuff in qemu too (well ig I could also implement virtio net or whatever)

[hot cobalt]

i read that wrong and i thought you somehow had an RTL8169 with MSIs! that would've been a novelty

[cyan slate]

do u support all resource types?

oh i see ur doing a switch case thing

for large resources with variable fields this is gonna create insane code duplication just warning u

[molten wyvern]

most of them look to have the variable length stuff at the end tho? so it would just be a memcpy and then a little stuff at the end

well ig I'd still have to duplicate the memcpy + the length check but eh, if that's the only thing then I don't think I really mind

also I think I might consider changing the huge interpreter opcode handler switch to a manual function call table because the stack usage of that function is over 4kb even in release now and considering it can be reasonably be called two times nested it needs at the very least like 9kb of stack to be reasonably safe to use

[cyan slate]

well idk try and find out

[molten wyvern]

I partially implemented tcp sockets, so now I can both connect to a tcp server and receive/send data and also function as a server too (which works good enough that I can serve a 404 web page). I haven't yet got disconnecting to work properly tho

after I fix that I kinda want to make a remote music player

and well I should also implement udp sockets and dhcp so that I don't have to hardcode the ip

[molten wyvern]

I implemented udp sockets, dhcp and also fixed the tcp disconnecting logic

dhcp in a kernel is kinda meh but I don't really want to implement it in userspace right now because it would need raw sockets

[ivory cloak]

so do you plan on implementing any sort of disk drivers soon

and fs drivers

[molten wyvern]

now one thing which I forgot about tcp that I should do something about is disconnecting the sockets when the pc is shut down or rebooted because else if you eg. listen with netcat for logging over network you need to also restart netcat (because it doesn't know the pc's connection died)

[molten wyvern]

I should do that lol

I kinda want to try ahci because I haven't yet tried it

[ivory cloak]

even obos at one point had them (they were shit), you don't won't to be worse than OBOS, do you? /j

[molten wyvern]

my c kernel also had nvme + ext4 (without writing or journaling)

[ivory cloak]

For OBOS, I hope on getting some sort of drivers for this sometime soon (after I exorcise the kernel, that is)

[molten wyvern]

also I want to try making loadable kernel modules

[ivory cloak]

that's one thing OBOS had

I was very proud of it

(then I rewrote)

anyway they aren't too hard

hardest part is the elf dynamic loading part, idk about unloading because I never implemented that

my kernel's code for that was... questionable

but I shall stop littering your thread now

[molten wyvern]

and ig if you want to restrict the amount of symbols exported then it would be harder too

[ivory cloak]

what I did is I had a macro called OBOS_EXPORT which, well, exported symbols*
*It marked the function signature as weak, because then the linker would put the symbol in the reloc table when it's referenced. I don't know if that's reliable or not though

then I slapped them on some functions in the kernel

I mean there is also modules interdepending on each-other

I haven't had to deal with that

[molten wyvern]

anyway Ill have to think about what exactly I want to do next now that I have fixed the most obvious bugs that I had

well maybe I should finally implement a load balancer so the other cpus have something to do other than idling

(and also look up all the conditions that I had though that it must respect + fix some code that isn't thread safe with "real multithreading")

[winged parrot]

thers visibility(export) lol

or whatever its called

[ivory cloak]

(which I recently found out about)

[winged parrot]

__attribute__ ((visibility ("default")))

[ivory cloak]

so qwinci you say you're good at debugging

are you so good that you can exorcise my kernel

(which I'm pretty sure is impossible)

[molten wyvern]

what is the problem

also maybe in some other channel than here lol

[ivory cloak]

move to obos channel

[dire galleon]

what I like to do for the kernel is set the visibility of everything to hidden, and then mark functions I want to be visible to drivers with default. You can extract that info from the symbol table later.

[molten wyvern]

I started implementing a load balancer and it worked fine for a couple seconds and actually improved the responsiveness of the desktop thing that I made but then it got stuck and I am pretty sure that might have something to do with the same thread potentially being executed on two cpus at the same time, that's one of the reasons why adding the previous thread to the cpu's run queue after having fully switched to the next thread is a good idea

[molten wyvern]

looks like it wasn't caused by that, this is one of the things that I don't really like debugging because it involves multiple cpus and pretty much everything scheduling related

[molten wyvern]

I just got rid of the code for now, Ill try rewriting it and going through all the sched code and thinking about all the possible cases

then also apparently there is some sort of gcc skill issue that causes __cxa_pure_virtual to get called when compiled without optimizations (with opts it works fine and on clang it works fine both with and without opts)

[molten wyvern]

apparently it was an unrelated bug related to threads sleeping and my event implementation

now Ill have to implement tlb invalidation and also fix aarch64

[molten wyvern]

I finally implemented close buttons for the windows and also killed performance in the process because I removed all the clipping/dirty rect stuff, Ill have to rewrite that too

[molten wyvern]

I just discovered a weird bug

(the video contains a little bit of flickering)

[molten wyvern]

apparently it was a kernel bug in the ipc socket where it could read more data than the supplied kernel buffer could hold (if the socket has more data buffered) and also copy that much data to the userspace buffer so it was both a kernel buffer overflow and an userspace one (the buffer that I passed to the syscall was on the stack so then ig it corrupted the stack)

[molten wyvern]

I implemented clipping + dirty regions for the desktop and also added a "taskbar" (the colored buttons on the bottom, though I have yet to actually wire them up to the windows)

there are some minor bugs with the whole thing though like I have yet to add a way for the window to tell it needs an update, the primary reason why I didn't do that rn is that I need to somehow get the correct response sent by the wm to the application from a stream of other response/event messages or alternatively have some kind of separate ipc channel for control stuff

and the buttons sometimes don't properly get redrawn if you hover over them with the mouse button down (which I am pretty sure is because the mouse leave events on them aren't properly called in that case for some reason)

[molten wyvern]

I fixed that so now the only remaining thing is the update thing but I am not sure how would I do it, maybe I could use a pipe for the events and then leave the main ipc channel for control stuff

basically it would be that first the application connects using an ipc socket and then the wm creates a pipe, moves the read handle to the other process and sends that handle to the other process over ipc as the first thing it does so the app can then use that to read the events

[molten wyvern]

I did that and also fixed all of the aarch64 bugs so now there are no known bugs anymore so ig now I can also work on making the aarch64 port better (as it lacks a lot of things)

or making a proper driver interface and loadable drivers, that's something else I have been wanting to do for some time

[molten wyvern]

now the aarch64 port is about as good as the x86_64 port as in that it has pre-emption working so now I can run the desktop thing I made on my phone it looks really funny tho

one problem is that the smp init sometimes causes the phone to crash and I haven't been able to reproduce that in qemu (and I have a feeling its some reserved memory being written or alternatively that the ap executes some garbage with mmu disabled before the exception handlers are setup but idk why wouldn't that always happen)

ig I could try it on the raspi5 too to see if it would be reproducible there as I can debug it with a debug probe unlike the phone

[molten wyvern]

looks like it works better after I modified the ap startup asm a little by adding a couple isb's before and after initially setting sctlr_el1 because apparently eret might not be enough to synchronize sysreg writes

though its going to take a while to see if that actually completely fixed it because it didn't happen that often

[dire galleon]

honestly thats pretty cool, nice work

[molten wyvern]

looks like it still does 💀 I really hate that it doesn't provide any kind of crash reason (or well maybe there is a reason somewhere within the 6gb mem dump that I can get out of it but like nowhere easily accessible like printed to the screen)

[molten wyvern]

now like 3rd try in improving it because apparently when I built the kernel in release then it crashed every time because for some reason the ttbr1 that I tried to get from a global variable in the code where the mmu is disabled was garbage (idk why it wasn't on qemu tho)

I really hope that this is the same issue that it sometimes had on debug builds

[molten wyvern]

now there is a proper xhci driver + hid mouse driver (though its not the best because the input may get delayed by like 50ms at worst, this is why I need some way to have driver threads that don't get preempted and that can be scheduled as soon as possible over any non-important other threads)

but other than that I am pretty happy with the usb interface in general and it should be possible to implement it for the other older controllers too (when I get around to implementing them lol)

now ig the next step would be to add keyboard support (which is mostly just a matter of adding new ifs for the keyboard hid usage page), maybe more mouse buttons because rn it only supports 3 and then maybe rndis or whatever so I could get "wireless" network (well not really wireless but pretty close as then I could just have my phone plugged into my laptop using usb)

also I forgot to kill the usb driver thread when a device is disconnected

[still lotus]

how will you handle locking in lower priority threads which want to access these data structures and high priority threads that need to write to it with little to no latency?

[molten wyvern]

yeah that's a good question, for an example rn I just lock a normal mutex protecting the event queue inside the driver thread but that might cause the thread to take a long time before its scheduled again if some other thread has the lock acquired at that time. ig unlocking the mutex could check if there is a higher priority thread and then switch to it or whatever

idk if that's what you meant but this is something that Ill have to think about if I am going to do that

well I pretty much have to do something about this because with eg. network 50ms of latency is pretty big in addition to the latency that the network itself adds

[still lotus]

I was thinking if you got interrupted while holding a preemptable lock and then ran something else in the meantime, you‘d get the delay just described

a fix would be to not allow preemption while holding this lock (i think that‘d fix it?)

but i was just curious if you had some well thought out locking strategy, which i would have been interested in : )

[molten wyvern]

yeah no I don't lol

I just disable interrupts before acquiring the lock

[still lotus]

but it works so that’s fine

[molten wyvern]

doesn't solve the other delay problem where if the lock is being held by an other thread while it tries to acquire it tho (as then it just switches to some other thread potentially for 50ms even if the other thread unlocked the lock sooner)

[still lotus]

does your lock give up the cpu if it’s not free immediately?

[molten wyvern]

yes, though it would certainly be better to spin for a while

actually that would probably fix it because its not like the lock is being held for that long at a time

[still lotus]

yielding for an interruptible thread immediately isn’t necessary is it

since you can’t deadlock on the same core because it‘s not preemptible

[molten wyvern]

yeah its not

[still lotus]

also 50ms for a threads quantum seems quite extreme

or am i misunderstanding something

[molten wyvern]

well its mostly just threads that don't block at all that get 50ms, originally the thread gets like 4.1ms and then it gets bigger if it just keeps on consuming cpu time (and at the same time it gets lower priority because the threads with lower slice time are preferred)

so if you'd have eg. while (true) asm volatile(""); in a thread then after a while it would get to the lowest priority level and be given 50ms at a time (assuming there are no other higher priority threads)

[still lotus]

could you give the thread holding the lock a higher priority so it unlocks it "earlier"?

[molten wyvern]

maybe, though I don't think it would really affect it because it only holds the lock with interrupts disabled

[still lotus]

oh yeah i‘m stupid

You could also reschedule after receiving the the mouse interrupt so the driver‘s allowed to run

[molten wyvern]

that could actually help

[molten wyvern]

now an initial rndis driver is done, it wasn't bad at all as its like 450 lines compared my basic rtl8139/8169 driver that's 500 lines

(and I also made some general improvements to the xhci driver like making it more thread safe)

didn't fix the driver delay issue yet tho, ig that's something for tomorrow

[hot cobalt]

how does your xhci driver compare to qookie's start of the art?

[molten wyvern]

in what aspects? I am not that familiar with the managarm xhci driver

[cyan slate]

It generally doesn't work

[molten wyvern]

at least there are def some completely missing things like hub support and likely some other minor details like that

[hot cobalt]

i should like to do an xhci driver butit seems a bit intimidating

there's no way i could pull it off in a few days so i would need a strong motive

[molten wyvern]

I found the usb book to be of great help

(along with the spec ofc)

[cyan slate]

Yeah if you want proper support for the entire stack its probably at least a month worth of full time work

[molten wyvern]

I don't really know how well my driver works, I have only tried it on my laptop (with an old mouse it doesn't work at least, though I am not sure if it even is hid compliant and with a keyboard at least it works enough that it sets the config so the leds turn on) and qemu

[cyan slate]

Yeah its probably hid compliant

[molten wyvern]

yeah looks like it is, it generates a transaction error when I try to get the first 8 bytes of its device descriptor initially no matter how many times I replug it (maybe its something to do with it being low speed instead of fullspeed like the keyboard is)

[karmic coral]

:(

[cyan slate]

Lol

I mean some improvements have been made recently ofc

But I have yet to get any of my devices to work on any hw

[karmic coral]

send me all of your hw and i will make it work

[cyan slate]

Lol

[karmic coral]

that reminds me

there was one thing i wanted to check in linux

[molten wyvern]

looks like it sometimes works to some extent at least if I have it plugged in from boot but not at runtime (and even with the keyboard it sometimes doesn't work at least if its already connected on boot)

I also tried different packet sizes on ep1 but it didn't help

[dim current]

Most epic music for a windowing system demo

[molten wyvern]

lol

[molten wyvern]

I had to try to make a thing that downloads a wav file from a http server and then plays it now I just have to turn this into some kind of proper thing you can give urls to (as rn its just hardcoded to some random beep wav on pitust's domain)

though the thing is that its not that useful because a lot of random wavs you find on the internet are on servers that require https

(unless its your own ofc, then you can make it work)

[molten wyvern]

I think I am going to switch the libc in crescent to the more proper one that I made, maybe make some sort of posix compat thing (that is pretty much needed because the libc needs that stuff) and then port something because I think that would be fun too

[molten wyvern]

I added a bunch of misc things like legacy pci support, power button + ec support but now I think I am running into some sort of scheduler issues the mouse lags a lot on the desktop thing that I made and the kernel discards a lot of the mouse events because of the event queue being full, its "fixed" if I make sys_poll_event use no timeout but then its going to be basically continuously polling for the events which isn't good for cpu usage

I haven't even touched the sched stuff in like 2 months, idk when exactly has it started to be like that or if it has worked properly

[molten wyvern]

looks like its somehow the load balancer, if I disable it then a 7ms sleep is at max like 20ms but with it enabled the 7ms sleep can somehow be up to 1s 💀

turns out my sleep function was slightly broken in that if a thread would try to sleep for a longer time than any of the already sleeping threads then it would just get inserted at the beginning of the sleep list when it should be at the end so then the other threads would only be woken up after the thread with a long sleep time as its supposed to be sorted

[molten wyvern]

I also rewrote the ps2 stuff based on linux i8042/libps2 source so hopefully it now works more reliably (and as its pretty similar to linux in basically all aspects except some c++ stuff and things like that so I added the copyright notices from there to be safe too)

[ivory cloak]

@scenic rampart look into this

[scenic rampart]

[cyan slate]

qwinci make a ups2

[molten wyvern]

I mean it wouldn't be that unreasonable in general, you could even make it generic over the serial with fn ptrs if you really wanted

[molten wyvern]

I finally added psf text rendering to the desktop thing, its very laggy tho because it generates the glyph bitmaps for every used char each time it needs to redraw something (I should just cache them and only generate if the char is not in the cache or the size is different)

what I actually wanted it for was not window titles but rather the start menu so I can generate a list of buttons with executable names in them that you can use to launch executables (as rn there isn't any way to do that)

then also for that I need a readdir-like syscall but it shouldn't be too bad at least for the dumb vfs I have over a tar archive lol

[dire galleon]

That's cool

The performance seems alright too

[molten wyvern]

in general it is but you can notice the lag that the text rendering causes when you try to drag a window with a title in fast circles vs without it (its not really noticeable in the video)

[cyan slate]

qwinci

are u going to implement a shader compiler and stuff yourself?

since this is fully nih

or porting mesa is possible

[molten wyvern]

I haven't really though about that lol, on the other hand stuff like that could be interesting to implement but it would take a ton of time and very likely end up being worse than what's in mesa

[molten wyvern]

maybe its now time to finally switch to hzlibc, though I changed the plan a little so I am only going to use it with the ansi part + rtld at least for now

[molten wyvern]

now that I did switch to it somehow the whole desktop thing became more laggy 💀

I don't understand why, it doesn't even use any libc functions except malloc/free and I used the exact same malloc impl in the garbage libc that it had and the better one

actually looks like its not the libc, it was me enabling stack protector because I though it would be nice (it was previously disabled because I didn't implement any of the stuff it needed)

ig Ill just keep it disabled then

[molten wyvern]

apparently it somehow got slower again even tho I didn't even change that much anything except adding support for aarch64

[molten wyvern]

I found at least one thing and its that the libc memcpy is garbage when not built in release (its literally just a byte by byte copy because the better assembly one that I made with movnti doesn't compile in debug using gcc, I should just write it entirely in assembly so it wouldn't rely on the compiler to allocate registers), exchanging that to a rep movsb one made it a lot better though its still not at the same level as it was before I did all this and with smp its a lot worse

[molten wyvern]

I really hate debugging performance issues lol

they are somehow even worse than normal bugs

I could just ignore them forever but it doesn't make a very good experience if you'd actually want to use the thing, like if you move a window around fast with a key pressed then you might get multiple seconds of input lag and eventually completely discarded events if you continue doing that for a while

a proper profiler could be nice but idk how you'd actually do that, I have used perf with the desktop thing in userspace but these issues likely have something to do with the kernel and the os-specific parts

[winged parrot]

you know you can do perf on qemu right?

might be a bit sketch but...

[molten wyvern]

idk if that's very useful for analyzing kernel performance issues tho

I mean maybe you can get something out of it if you are lucky but in general it sounds painful

I think I might just try to incrementally apply the changes from the last commit on top of the previous one that had better perf and then see if there is some individual change that is causing it

would be nice to split up the commit anyway, I was too lazy to do it properly so it has some misc changes I made to implement the libc sysdep functions along with eg. changing the thread startup to conform to sysv + saving rflags on context switch (to preserve interrupt enable status)

[molten wyvern]

looks like it mostly has something to do with the libc, I don't understand what tho

maybe its just the fact that malloc/memcpy/etc calls go through the plt? but idk why would that add so much lag

I should make the new libc support static linking and then try if that would make it better

[molten wyvern]

apparently there are even more issues with the sched thread unblocking stuff too, for some reason there can be a thread in the sleeping thread queue with its status set to waiting which makes no sense

or actually wait maybe its not in the sleeping queue anymore at that point

unblocking a sleeping thread is annoying

the way I do that right now is that to call unblock you have to hold the thread's sched lock to prevent it from being moved to a different cpu while its being unblocked and then inside unblock I acquire a lock on the thread's current cpu's sleeping threads list, that's the problem because the code inside the sched timer handler does it in the opposite order first acquiring the sleeping threads list lock and then thread lock potentially causing a deadlock

I guess I could add a precondition that you must also have locked the sleeping threads list before locking the thread lock but that's kinda stupid because its not needed if the thread isn't sleeping

and now that I think about it that's impossible to do because to access the sleeping threads list you have to go through thread->cpu and to safely access that you need to hold a lock on the thread

I think this is precisely why I avoided directly messing with the sleeping threads list in the past and instead set the thread's sleep end to SIZE_MAX and then let the sched timer code handle removing the thread from the list

that approach has its own issues too, for an example I would have to iterate through the whole sleeping threads list every timer interrupt instead of being able to stop on the first thread that has a sleep end after the current point in time

so yeah idk what should I do about this

[molten wyvern]

I just ended up doing that not ideal approach of iterating over the entire list for now and reverted the libc change, Ill try to take an another take on it after it supports static linking

[molten wyvern]

I added a directory list syscall and entries for the files in /bin to the start menu, now I just have to hook up the create process syscall to the clicks and its hopefully going to just work™️

Ill also have to make some sort of scrollable window at some point so the menu can have more entries than what can fit into it but that's for later

then after that idk what I want to do next, ig I can also add reboot/shutdown buttons to it and then one thing which I have been wanting to do is rtc clock

and maybe even sleep too it sounds kinda annoying tho because you might resume in real mode

[scenic rampart]

Wait wtf

What

I know nothiung about sleeping does it do that

[molten wyvern]

it depends on what the firmware supports, either you are going to wake in real mode, protected mode or if the firmware supports it you can request to be resumed in long mode

ofc you are (hopefully at least) going know what mode you are going to wake in before you go to sleep so you can point to the right wake code that does the necessary stuff

[scenic rampart]

That sounds super annoying

Does all processor state just get reset?

[karmic coral]

afaik S3 sleep is basically shutting down except the ram is kept powered on (+ whatever to keep it refreshing)

[molten wyvern]

yeah

and then ig you have to do ap boot up again too

and there is no limine to help you at that point

[scenic rampart]

Damn

[cyan slate]

there are different sleep states

for s1 and s2 u resume where u left off

s3 u resume at wake vector in real mode

s4 is a from scratch boot, ram is gone

and s5, thats just full shutdown

[scenic rampart]

Ohhhh that makes sense

Thank you acpi man

[cyan slate]

np

[ivory cloak]

What happens with hardware other than ram

Like ahci controller

And drives

[cyan slate]

whatever u do to it

it wont autoshutdown

u have to shut it down yourself

[ivory cloak]

Oh ok

I remember there being power control stuff in the ahci spec

[cyan slate]

yeah

all pci devices have D states

D0 D1 D2 D3Hot d3Cold

https://learn.microsoft.com/en-us/windows-hardware/drivers/kernel/device-sleeping-states

[molten wyvern]

so you don't have to reinitialize the devices if you don't do anything to them after waking from s3?

From S3 Sleeping State, the wake event repowers the system and resets most devices (depending on the implementation)
well maybe you do

[cyan slate]

yeah

[cyan slate]

well this u need to be able to wake up

u detect which devices have wake capability and enable their wake GPE

[molten wyvern]

ah is it specifically referring to devices with wake capability? though why would it reset them if it doesn't do that for others

[cyan slate]

ye

i dont really have any deep insights since i never implemented s3 wake

in any kernel

its just a lot of work

[molten wyvern]

yeah

I still kinda want try doing it at some point, it could be cool

same for cpu power state management

[cyan slate]

yeah

u can start with qemu, it supports all of those states

[molten wyvern]

I added the process spawning thing and reboot/shutdown buttons, apparently it isn't a good idea to launch two instances of the desktop idk why does it literally kill the kernel tho (along with first creating some graphical corruption but that's expected)

[molten wyvern]

I bet its some kind of memory corruption but idk where

[molten wyvern]

I think its my pmalloc, the buddy logic is likely what's causing it because it depends on pretty fine locking and adding a giant pmalloc/pfree lock seems to fix this

I might just keep that giant lock for now because it looks kinda annoying to fix while using per freelist locks + per page locks

and I really need to only allow exclusive access to the framebuffer so you can't spawn more than one of the desktops at once lol

then after that maybe Ill look into the sleep stuff

[molten wyvern]

or well it also needs a smp bringup code that doesn't suck so it works on real hw so eh I might try the rtc stuff first

then I should also fix the intel gpu accelerated page flipping code at some point

[molten wyvern]

time is there, though Ill have to add some kind of way to adjust the timezone taking into account dst's and stuff

and I should separate this gui thing into a separate library because it's just a generic thing that works on top of a framebuffer so you could use it inside an individual window too

[molten wyvern]

• I should think about how this works on a phone, its not really going to work that well as it is right now

it would be an another reason to separate this stuff into a library so then I could make a separate thing specifically for devices like that without needing to duplicate that much code

[molten wyvern]

I am trying to do the sleep stuff but I really hate real mode, you can't even properly debug it

[cyan slate]

like _S3?

[molten wyvern]

yes

[cyan slate]

i see

real mode is easy to debug with bochs btw

[molten wyvern]

using the builtin debugger?

[cyan slate]

yup

u can single step real mode code

and add breakpoints with xchg bx,bx

[molten wyvern]

looks like even without that it printed the reason (though not the exact address)

[cyan slate]

yup

[molten wyvern]

idk how it resumed immediately tho lol

[ivory cloak]

Ngl this kinda makes me want to implement suspend

[cyan slate]

u probably dont quiesce all interrupts

[ivory cloak]

Via S3 states

And lower

[cyan slate]

so u get like a timer interrupt or something

[cyan slate]

lower is suspend to disk

and below that is shutdown

[ivory cloak]

Higher

I meant

[cyan slate]

ah

[ivory cloak]

By lower I intended to say S2/S1

[cyan slate]

its usually categorized as shallower/deeper

[ivory cloak]

noted

[molten wyvern]

ah wait yeah it wakes on normal irqs, I don't do anything to them so it makes sense (or well I do disable timer but there are others like the mouse or keyboard)

[cyan slate]

ur supposed to disable everything and only leave GPEs of the devices that support wake

[molten wyvern]

how does wake on keyboard/mouse press work tho, I don't think it has a gpe?

[cyan slate]

it does have a gpe if it supports wake

[cyan slate]

https://uefi.org/specs/ACPI/6.5/05_ACPI_Software_Programming_Model.html#determining-the-system-wake-source-using-wxx-control-methods

and this https://uefi.org/specs/ACPI/6.5/05_ACPI_Software_Programming_Model.html#managing-a-wake-event-using-device-prw-objects

just scroll up from there ig

[molten wyvern]

Ill take a look

[cyan slate]

lemme find how linux does it

[molten wyvern]

I don't see a gpe on my laptop's ps2 acpi device tho, it only has _HID/_CID/_STA/_CRS/_PRS (and the ps2 device is not mentioned anywhere else either)

yet it wakes from sleep when you press a key

[cyan slate]

https://elixir.bootlin.com/linux/v6.11.5/source/drivers/acpi/scan.c#L1031

[cyan slate]

it could be the i2c device doing the wake

doesnt have to be routed literally from ps2

[molten wyvern]

ah yeah maybe it is

[cyan slate]

check which devices have _PRW on your laptop

[molten wyvern]

there are like 100 of them lol

[cyan slate]

yeah there u go

[ivory cloak]

would it be a good idea to tell all of them that they can wake me on GPE

[cyan slate]

not that simple

each device tells u the range of states it can wake from

and the state it can be in to do it

uacpi has helprs for retrieving all of those

[ivory cloak]

k

[cyan slate]

Look at _SxW etc

where x is the respective state

anyways just read the linux source i linked

[molten wyvern]

I fixed the trampoline code, it was some att order memes

Ill have to do the other stuff properly tho because it looks that eg. the hpet is stopped so Ill have to start it again

[ivory cloak]

what are these helpers called

nvm I think I found it

[molten wyvern]

I wonder if I should have some sort of generic kernel device structure that would have suspend/resume functions and a list of all devices

rn I just have lists of user-exposed devices but even for devices that aren't exposed to userspace it would be nice to keep a list of them so I don't have to special case eg. hpet resume

[molten wyvern]

idk what should I do to sleeping threads tho, do I just restore the hpet counter value and pretend like no time passed while sleeping or do I actually get the amount of time from somewhere like the rtc and add that to the counter

[molten wyvern]

apparently the io apic is also reset I quite literally have to reinitialize almost everything except that I should reuse the cpu irq vectors (and other allocated things)

then also what I am interested in is what happens to the gpu and framebuffer on real hw

Ill have to test that

[ivory cloak]

I have a suggestion

have it so drivers can register some sort of callbacks that the kernel calls before/after suspending

so that they can shutdown/reinitialize their device

[cyan slate]

did u figure out your own resume

[hot cobalt]

how goes Crescent anyway

[cyan slate]

fully nih going as per usual

[hot cobalt]

i know there's a qlibc and a qacpi

there used to be a qloader2

[molten wyvern]

qlibc or qloader2 aren't related tho, they aren't made by me

[cyan slate]

wait qlibc is not yours?

arent u using it

[molten wyvern]

no

the one in crescent is a garbage one that I specifically made for it that barely contains anything at all

[cyan slate]

ohh

[molten wyvern]

then I do have the better hzlibc one but it killed the performance of the desktop thing so I reverted the change until I figure out what's causing that

[hot cobalt]

oh, it must have been renamed

i thought you invented qlibc

hzlibc: the libc that won't tick you off

[ivory cloak]

the kernel will have sleep/wake callbacks, sleep taking in a uacpi sleep state and wake not taking a sleep state. kernel modules will have them in their function table, and any sorta driver that's in the kernel (like anything to do with the lapic) will be able to register these callbacks through some sorta function

[cyan slate]

does it work now tho

[molten wyvern]

also I did kinda do that with a device struct that devices would inherit and then the driver registers the devices with the kernel but I am not sure if it would make sense to have it as a standalone thing so you could easily register multiple callbacks (well ig it doesn't really make that much sense)

[hot cobalt]

damn, hzlibc is a proper libc, i didn't realise it was so expansive in scope

[molten wyvern]

for the most part it is yeah, it also borrows some parts from mlibc (and musl for the math stuff)

[ivory cloak]

nah it's more like

void(*sleep)(uacpi_sleep_state going_to);
void(*wake)();

that applies to any device that the driver controls

[hot cobalt]

borrowing sun libm is fine, they are the only people who ever wrote a libm in the past 35 years

[ivory cloak]

and then anything else can register these as well

[molten wyvern]

ig that would work too as long as you keep some sort of list of the devices within the driver

[ivory cloak]

I mean usually you do in some way

like an ahci driver you have a list of the ahci ports

[cyan slate]

u dont really need to keep any lists

u have buses

which have devices attached

nothing exists alone

well the bus would have a list internally ig

[clever shell]

croissant

[molten wyvern]

I guess I should also call the suspend function basically for all devices? because I can't really know which devices would be usable after the wakeup

[lament drift]

https://tenor.com/view/jimmy-neutron-carl-wheezer-are-you-going-to-finish-that-croissant-croissant-cwasont-gif-21829791

literally

[clever shell]

https://tenor.com/view/national-croissant-day-croissant-national-carl-wheezer-carl-wheezer-gif-gif-24705602

[molten wyvern]

lol

[ivory cloak]

I guess

I mean not all devices really need to do anything before suspend

but most, if not all are basically gonna be dead after you're awaken

[molten wyvern]

yeah

[ivory cloak]

although you might need to set some devices to be initialized in your trampoline

e.g., before you call into any sort of drivers that use your scheduler, you'll probably want to get your scheduler back up

so you'd do that in your trampoline

[molten wyvern]

the thing is that the current code for initializing the lapic depends on the hpet so unless I reuse the old calibration Ill have to have hpet be restored before that

well ig the lapic speed doesn't likely change that much so it might make sense to just skip the calibration

[ivory cloak]

so then the hpet also gets thrown to the list of devices that you wake manually

in the trampoline

and you do it before the lapic

[molten wyvern]

maybe

[molten wyvern]

ap re-bootup is done now, it faults when entering the idle thread tho (even on the bsp) for the ap's it would kinda be expected because they might be in the idle thread when the cpu is stopped but for the bsp there shouldn't be anything corrupting the saved state

[molten wyvern]

there is also some weird bug where a kernel virtual space allocator regions list spinlock is deadlocked sometimes, I have no idea how that can happen because its always acquired with interrupts disabled

[molten wyvern]

it only happens like once in a few tens of qemu reboots if at all so ig Ill leave it be for now because even when I get it there is no good way to see what exactly caused it (I also tried to store the last lock/unlock locations inside the lock and they don't match at all as the last lock was inside alloc but the last unlock was in free)

[molten wyvern]

actually I think I figured it out, I used gdb to set the spinlock to unlocked state and then when it switched to the next thread it looks like it had been waiting on an allocator mutex when freeing the region struct while still holding the regions spinlock (the fix for this should just be to make sure that the spinlock is unlocked before the region is destroyed, it doesn't really make sense to hold it at that point anyway)

now I can get back to figuring out the resume problems ig

[molten wyvern]

ah right I execute the actual sleep code on a cpu that is potentially not the bsp, that explains why the bsp's idle thread gets corrupted

[molten wyvern]

I tested the kernel in general on real hw again and apparently I sometimes get a power button notify right after the kernel has started and I have no idea why lol

it just shuts itself off right after it has started because I have it shutdown on power button press (and that only happens sometimes, like I think after a forceful shutdown it doesn't happen for an example)

at least that way you can't possibly run into any bugs that would occur later in the drivers

[cyan slate]

Maybe you run some aml incorrectly which causes that

Is it an actual notify via a method?

[molten wyvern]

I am pretty sure it is a notify op that you get after an ec query method is run yeah

[molten wyvern]

I can't even reproduce it anymore, idk what were the things that I did

[cyan slate]

test managarm on it

[molten wyvern]

I mean it works fine, I already know that

but so does crescent most of the time so idk what exactly caused that to happen multiple times in a row

[cyan slate]

ah

[molten wyvern]

I finally got around to implementing ramfb for aarch64 so you can have graphical output on qemu there is no keyboard or mouse or anything tho so its pretty useless in that sense

[karmic coral]

just need pcie and xhci :^)

and a usb stack

[molten wyvern]

I already have the other parts than pcie, I haven't yet looked into implementing it on aarch64

the xhci driver is slightly broken on real hw tho but at least last time I checked it still worked in qemu

also other things which I want to do is improving the acpi sleep by properly enabling the wake devices and adding suspend/resume handling to the remaining devices, maybe supporting sleep on aarch64 using psci suspend, separating the ui thing to a library so I can make some kind of better desktop for small screen devices (and also ideally automatic layout because manually placing everything relative to each other kinda sucks), maybe support for the touchscreen that I have on my phone if its not too bad and then there is the dwc3 xhci controller that would be nice to figure out how it works so it could be made work on managarm too

idk if the dwc3 requires using resets or whatever tho, if it does then on managarm it would be best done after the other dtb stuff that you were talking about (exposing stuff like reset assert/deassert, clocks and regulators)

[molten wyvern]

I should bisect the hda issue on the laptop too as I am pretty sure astral has the exact same issue and I know that it worked fine before I changed the hda driver around while exposing it to userspace (I just verified that and it indeed does work fine for both the speakers and the headphone jack with the old version of it)

and I don't want to transfer that bug to managarm too lol

I wonder if putting the hda stuff to a uHDA-like library would be a nice idea (obviously expanded from what I have done so far to allow more use-cases), it would also mean that everyone using it would benefit from fixes and other additions

[cyan slate]

yeah it would definitely

[molten wyvern]

I figured out the issue that I had with hda on my kernel, it was that when I modified the old code I forgot to readd the code setting the stream format that doesn't explain the astral issue tho because there I did remember to do that

maybe if I write that library I can manage to not make whatever mistake I made and then it would just work™️

[molten wyvern]

I wonder what should I name it tho, do I literally just name it uHDA lol

[cyan slate]

lol

that would be funny

[lament drift]

someone should make an EFI library called uEFI

oh wait

[ivory cloak]

someone should make a driver interface

for hobby osdev

uDI

[scenic rampart]

uHCI

[ivory cloak]

uKernel

[molten wyvern]

lol

[scenic rampart]

uBitches

[ivory cloak]

[ivory cloak]

no uBitch

uGPU

uPS2

uAHCI

uFAT

[molten wyvern]

anyway I think if I get the hda lib to work then I can switch the astral driver to that (though it would need retesting because I have limited amount of devices to test it on, basically my smartsound laptop + some laptop that I got for work + my desktop)

[ivory cloak]

uEXT2

[scenic rampart]

uBerrow

[ivory cloak]

O_O

[lament drift]

uRMOM

[ivory cloak]

uEGirl

[molten wyvern]

uTouch

[cyan slate]

what does this do? @molten wyvern

[molten wyvern]

its a workaround, basically in aml I don't evaluate fields until they are actually used somewhere where eg. an integer is required so then there can be eg. packages that contain field objects being returned from methods that you manually evaluate and you want the pkg element to be an integer/buffer and not the field

[cyan slate]

can u show an aml example?

[molten wyvern]

OperationRegion (ECMM, SystemMemory, 0xFE0B0800, 0x1000)
Field (ECMM, AnyAcc, Lock, Preserve) {
    Offset (0x5D), 
    ERIB, 16
}

Method (FOO, 0, NotSerialized) {
    Return (Package () {
        ERIB
    })
}
``` then you evaluate FOO manually and it returns a package with a field object

[cyan slate]

shouldnt it not read from ERIB here

iirc it should just be an "ERIB" string object

[ivory cloak]

delusional aml moment

[cyan slate]

like i dont think NT cares which type of object it is, it never resolves it

in uacpi it certainly wouldnt read it too unless im rarted

[molten wyvern]

idk

I could certainly read it from there if its an object that exists at the time the package is constructed

[cyan slate]

sigh

let me pull up my nt debugger

[molten wyvern]

ig that's one thing that I do differently, idk if it really matters in practice as I just try to resolve it and if it fails then its stored as unresolved

(and resolved on the first access)

[cyan slate]

well it matters in this case because reading from a field has side effects

[ivory cloak]

warning: debugging NT at 3 am can cause hallucinations

[cyan slate]

and like

no aml stores fields in a package

because theres no user facing api to read from an aml field

anyways lemme actually check

    OperationRegion(REG, SystemMemory, 0x100000, 128)
    Field (REG, AnyAcc, NoLock) {
        FILD, 32
    }

    Local0 = Package {
        FILD
    }
    Debug = Local0

this will be my test case

or actually let me move it into a method so i can test on acpica as well

ACPI Debug:  Package  (Contains 0x01 Elements):
ACPI Debug:     (00) 0x0000000000000000

acpica does read it

fuck, i got a BSOD

let me change a few things around

[molten wyvern]

also an another question is that in the case its not read (or can't be resolved at that time so its impossible to read) should it only be read once when its accessed and then stored within the package or is it read once on every access to the pkg element

tbh I'd guess the first option, the second one is kinda dumb (and I am pretty sure its also what I am doing)

[cyan slate]

no of course it shouldnt be read dynamically

or god forbid every time

[molten wyvern]

I think with the way I do it if you'd assign something else to the pkg element that field is in then it would also write to the field

[cyan slate]

wtf

i've learned that with the nt interpreter it does the simplest thing every time

no insane two pass like acpica

no crazy dynamic resolution of anything

yay I was right

uACPI agrees:

[uACPI][TRACE] [AML DEBUG] Package @0000603000001630 (0000603000001690) (1 elements)
[uACPI][TRACE] Element: String => "FILD"

and the thing is its literally never even attempted to be resolved

unless the client explicitly does it on their own

e.g. uacpi_object_resolve_as_aml_namepath in uACPI

[molten wyvern]

is it actually a plain string instead of some kind of unresolved object?

[cyan slate]

its literally a plain string object

in uacpi i tag it as UACPI_STRING_KIND_PATH

NT has no such thing as unresolved object

whatever path u put in that package, bogus or existing it will always store it as a string

[molten wyvern]

I'd hope that no aml relies on it being a string tho

[cyan slate]

who knows

on acpica it literally reads it right away too

like how wrong is that

idk if you've seen it in my thread or not, I have added this recently

[molten wyvern]

I don't think I saw that, yeah ig that makes sense

[cyan slate]

there's also a lot of aml code that just has random bogus garbage inside the package, since NT doesnt error out in that case they just leave it there

and then acpica chokes on it

im not sure if acpica leaves a NULL in that case or what

or maybe just dies

[cyan slate]

i tried to modify it via str[idx] = "a" but as u know the NT interpreter segfaults on IndexOp on strings

so i cant really check if its writable

but id imagine it is, since even the aml debugger prints it as a string object in that output

[molten wyvern]

I guess I am going to change my logic to match that, the current one is very weird

[cyan slate]

its just the simple thing to do tbh

btw u do survive CopyObject(IDK, \) \() right?

[bleak remnant]

infy
Compile Error! Click the reaction for more information.
(You may edit your message to recompile.)

[cyan slate]

lmao

basically turning _GL or \ into something else via copyobject

would your interpreter not crash in that case?

and if it does, what happens if I then write to a locked field?

[molten wyvern]

it fails with an error about invalid aml lol

[cyan slate]

lol

how is it invalid

[ivory cloak]

infy, isn't it 3 am where you are lol

[cyan slate]

my sleep schedule is shit

[ivory cloak]

@cyan slate go to sleep

[cyan slate]

u mean prepare_for_sleep_state

[ivory cloak]

hold on execute this code

[cyan slate]

does sleeping count as suspending yourself

[ivory cloak]

uacpi_prepare_for_sleep_state(UACPI_SLEEP_STATE_S3);
cli();
uacpi_enter_sleep_state(UACPI_SLEEP_STATE_S3);```

[ivory cloak]

yes

dying is S5

[cyan slate]

lmao

[ivory cloak]

sleeping is S3

coma is S4

[cyan slate]

S4 is coma

[ivory cloak]

S1 is being at work/school

[molten wyvern]

I am trying to see

[ivory cloak]

S0 is coding something and it working first try

[molten wyvern]

oh yeah it fails to resolve \ as its smaller than 4 and I forgot to remove that check when I moved the other resolving code to context

[cyan slate]

Lol

[molten wyvern]

qacpi: internal error in handle_name, node->object is null
internal error
executing MAIN()

[cyan slate]

F

[molten wyvern]

oh yeah \ doesn't have an object, that makes sense

ig I should just attach a dummy device to it

[cyan slate]

Well ObjectType(\) is Uninitialized

[molten wyvern]

ah ig Ill do that then

[cyan slate]

What about gl btw

[molten wyvern]

I don't see why that wouldn't work, its just a normal device

[cyan slate]

Well idk how you handle locked fields

But gl is a Mutex, not a device

U do support Acquire(_GL) right?

[molten wyvern]

oh yeah, I should go sleep lol

[molten wyvern]

yes but it doesn't do the proper gl stuff

[cyan slate]

Ah

[cyan slate]

Same

[molten wyvern]

actually Ill try what happens if you copy something not a mutex to gl

and then use a locked field

[cyan slate]

Well do your locked fields use the gl object?

[molten wyvern]

I think I removed the field locking code at some point, I didn't even remember that

that's something that I should readd and properly implement gl

[molten wyvern]

@cyan slate regarding the package/field stuff, what are the ways that aml can actually get the field value out of a field ref inside a package?

[cyan slate]

it cant

it cant get anything out of a named object inside a package

because its a string

[molten wyvern]

ah, that's actually good

I were thinking that if there are some cases where its actually supposed to do that then it could be painful to implement but that makes it very easy

[cyan slate]

yeah

basically NT treats packages as a strictly data passing interface, not a language feature

[molten wyvern]

I wonder what should I do to it in my get_pkg_element tho, ig I don't want to modify the pkg in place but then its going to read the field multiple times if you call get_pkg_element multiple times on the same thing

[cyan slate]

why is there a field object there

[molten wyvern]

actually yeah its probably an another issue related fields that causes a field to get inside a _PRW package

because in the aml it passes a field (or rather it should be the value of it) to a method that then puts that arg inside the package and with the way I do it rn the field object itself gets passed to the method and then gets put inside the package

[cyan slate]

ah

in that case it must be read at call site ofc

[molten wyvern]

yeah

though it would still certainly be possible to get a field object when resolving a path stored inside a package (idk if anything actually does that in practice)

[molten wyvern]

now that field stuff should be fixed, ig next I might look into _GL (though idk how do I actually want to implement that, I could just have kernel fns for writing to the facs but then again I need full table access for some of the remaining aml ops so it might just make sense to have that instead)

and I have to look more into the sleep stuff too, on my laptop it still kinda wakes up after a second but idk what state it is in (I tried to install the sci handler so I could press the power button to shut it down but that didn't work, idk if its even running the kernel code or not)

and the display is completely off

[molten wyvern]

I added resume support for ps2

now the remaining devices are basically rtl + xhci + hda

and then getting the fb back on real hw, I am not completely sure how do I do that (ig I can try the vga stuff that linux does or whatever)

[molten wyvern]

tbh I think I might take a slight break from the sleep stuff and see what would it take to get some touches

[cyan slate]

Touches?

[molten wyvern]

as in writing a driver for the touchscreen that I have on my phone

[cyan slate]

Oh damn

Which protocol is it

[molten wyvern]

spi

[cyan slate]

do u have it reverse engineered or?

[molten wyvern]

I just remembered why I don't like this stuff, there is all the configuration you have to do with clocks and gpios and whatever other stuff (or well idk if its all really necessary, depends on how much of that stuff the firmware configures)

[molten wyvern]

there is the linux source for it

and from the looks of it most of it is actually from a driver sample lol

some other stuff might have to be reverse-engineered tho like if I ever want to try using the modem as I am pretty sure that the kernel just exposes it as a serial that some userspace then interfaces with

would be based to send an sms from crescent tho I don't think I would trust it in normal use tho, what if I need to make like an emergency call and then it decides to assert(!"unimplemented")

[winged parrot]

dualboot android

[cyan slate]

Lol

Managarm for medical devices when

[winged parrot]

m4md

[molten wyvern]

I mean that could work, there are even the two sets of the boot partition that you can switch between

(and well you could always just boot using fastboot and be fine until you reboot the phone)

[winged parrot]

(you cant do that)

[molten wyvern]

[winged parrot]

android uses these partitions

for many important things

[molten wyvern]

the boot partition is only used for the boot image

[winged parrot]

yeah

important!

also, for OTAs

[molten wyvern]

and on A/B devices there are two sets of it

[winged parrot]

and there's no gui to switch

[molten wyvern]

it only uses the active one

[winged parrot]

you really shouldnt do that

just use a custom stub that does kexec

[molten wyvern]

does it actually boot it using the proper boot protocol in that case or how does kexec work

[winged parrot]

no idea

just write a stub that does dualboot how hard can it be /s

actually. how hard IS it to do that

you "just" need to do the buttons which is probably trivial

the screen which is just some bytes in ram

maybe timekeeping?

and storage which is the real big one

[molten wyvern]

basically like a bootloader?

[winged parrot]

yeah pretty much lol

i wonder if there's some really nice bug in my phone that lets you get untethered code execution in uefi

that would be cool

[molten wyvern]

the real question is where would that dualboot stub be stored tho

[winged parrot]

boot

[molten wyvern]

where would the real linux be stored then? or well ig it could be after the stub or whatever

[winged parrot]

cache?

or imagefv

[molten wyvern]

hoping that it doesn't get overwritten

[winged parrot]

or data

[winged parrot]

you have manual recovery for that

[molten wyvern]

it doesn't sound that bad tbh, even for the storage idk if its really that bad either, its "just" ufs + scsi (assuming its a phone with ufs)

and buttons at least on qcom aren't bad, its just slightly annoying because you have to go through spmi or whatever

[winged parrot]

the power button looks like pmic?

hm

maybe not on newer qcom

[molten wyvern]

it probably is, spmi is used to interface with the pmic

[winged parrot]

ah

[molten wyvern]

I think I have only tried the volume down button tho, its the only button that I found on the dtb for some reason (or maybe the others are just named weirdly or smth)

[winged parrot]

yeah seems like its all pmic

[karmic coral]

kexec boots it using linux boot proto afaik

[molten wyvern]

nice

[molten wyvern]

I started working on uHDA, now what I need to somehow figure out is how do I expose all the stuff in a good way

I mean I could always just have an api to configure/get the widget params and make the user figure out the paths + configuring the widgets themself

there are still some things I am not sure about like for an example there are like 6 different output pins on my laptop that are all grouped together and they are somehow used for multichannel

and there are two speaker groups where the other one only contains one output pin and then that other one with multiple, idk how are you supposed to choose which one to use (though if I had to guess I'd say that the singular one is used for regular mono/stereo and the other one for something else but idk)

[molten wyvern]

typedef void (*UhdaBufferFillFn)(void* arg, void* buffer, uint32_t space);

typedef struct UhdaOutputGroup {
    const UhdaOutput** outputs;
    size_t output_count;
} UhdaOutputGroup;

typedef enum UhdaStreamType {
    UHDA_STREAM_TYPE_IN,
    UHDA_STREAM_TYPE_OUT
} UhdaStreamType;

void uhda_codec_get_output_groups(
    const UhdaCodec* codec,
    const UhdaOutputGroup** output_groups,
    size_t* output_group_count);
void uhda_controller_get_streams(
    UhdaStreamType type,
    const UhdaStream* streams,
    size_t* stream_count);

bool uhda_paths_usable_simultaneously(const UhdaPath** paths, size_t count, bool same_stream);
UhdaStatus uhda_find_path(
    const UhdaOutput* dest,
    const UhdaPath** other_paths,
    size_t other_path_count,
    bool same_stream,
    UhdaPath** res);
UhdaStatus uhda_path_setup(UhdaPath* path, UhdaFormat fmt, UhdaStream* stream);
UhdaStatus uhda_path_shutdown(UhdaPath* path);
UhdaStatus uhda_path_set_volume(UhdaPath* path, int volume);
UhdaStatus uhda_path_mute(UhdaPath* path, bool mute);

UhdaStatus uhda_stream_setup(
    UhdaStream* stream,
    UhdaFormat fmt,
    uint32_t ring_buffer_size,
    UhdaBufferFillFn buffer_fill_fn,
    void* arg);
UhdaStatus uhda_stream_shutdown(UhdaStream* stream);
UhdaStatus uhda_stream_play(UhdaStream* stream, bool play);
UhdaStatus uhda_stream_queue_data(UhdaStream* stream, const void* data, uint32_t* size);

I think something like this might do, with some additional information ofc so you can actually know eg. whether an output is a speaker or not

this api would support both manually queuing data outside of irqs and then if you want to only queue data when needed you supply the buffer fill fn when setting up a stream and it gets called from inside the interrupt handler when there is space within the ring buffer

[molten wyvern]

@ivory cloak are you going to use uhda in obos if I get it working or do you want to write your own driver for hda

[ivory cloak]

is hda audio

[molten wyvern]

yes

[ivory cloak]

that seems like pain

so I'll be taking that

if you get it to work

[molten wyvern]

what I do hope is that its actually going to work on a decent amount of machines without needing the billion different hardware quirks that linux has because I don't feel like implementing all of those at least to start with

[cyan slate]

thats why we have the community here which is nice

they can test on everything

btw if u want we can come up with a design for a universal kernel api

thats (if enabled) will support both uacpi and uhda at the same time

[molten wyvern]

yeah that could be nice

[cyan slate]

id imagine hda doesnt need more stuff than acpi?

[molten wyvern]

I think the things that uacpi doesn't have are pci irq alloc/dealloc/enable/disable and just plain physical mem alloc/dealloc

[ivory cloak]

well mapping bars

usually they're UC

[cyan slate]

fair probably

the universal api must be a superset of both

[molten wyvern]

for reference this is what I have right now for the kernel api, the pci device void* could be replaced with an address so it could just use the same api as uacpi

[cyan slate]

yeah that seems pretty normal

why is it a void*?

[molten wyvern]

and yeah the pci bar mapping too as it needs the cache mode, it could either be done just by a generic memory map function which accepts the cache mode or a dedicated one (though the generic memory map function with uncached mode is still needed for other stuff)

[cyan slate]

bar mapping should probably be a separate one

because in advanced kernels it might lazily allocate resources and stuff

[molten wyvern]

there is no particular reason for that, its just an opaque handle to the pci device

[cyan slate]

like the kernel does the handle allocation or?

[molten wyvern]

yes it can pass whatever it wants as that and it will then get passed to eg. the pci irq alloc/enable fn

[cyan slate]

ah ok

[molten wyvern]

also one thing which I didn't add yet is a spinlock but its also a part of uacpi's apis

then for matching pci devices I am not completely sure what should the driver side api be, for now I just added a define with a list of devices (in the format {0x8086, 0xA0C8}, {0x1234, 0x1234}) and then a function that checks whether a vendor/device is in that list that you can alternatively use

that kinda depends on how the kernel side does pci device discovery, it might also be that neither of those really works in which case ig you would just have to manually copy the devices from the list and use the class/subclass match in whatever format your kernel uses

[molten wyvern]

it almost works™️ in qemu again (though tbf its basically the working driver that I had + some helper functions )

there is some sort of problem when you start playback tho but other than that

also had to change the api around so it takes triple pointers in some places because I didn't realize that you can't index arrays of incomplete types

[cyan slate]

#ifndef UHDA_PCI_DEVICE
#define UHDA_PCI_DEVICE(vendor_id, device_id, idk_something_else) (struct uhda_pci_device_id) { vendor_id, device_id, idk }
struct uhda_pci_device_id {
    uint16_t vendor_id, device_id;
};
#endif

struct uhda_pci_device_id hda_devices[] = {
    UHDA_PCI_DEVICE(0x123, 0x321, ...),
    UHDA_PCI_DEVICE(0x321, 0x123, ...),
};

maybe u could do something like this

[molten wyvern]

that could work

maybe I should also make the naming convention match uacpi

well basically the only thing it would change is that the public types would be lower_snake_case

[cyan slate]

I like that because its prefixed with struct anyway

Either PascalCase but typedefed or snake but with a prefix

[molten wyvern]

rn I use pascal case with typedefs, ig I could just keep it like that

still haven't figured out the bug tho, it sounds like it plays some 0xcbcbcbcb data for a moment and then restarts it from a little bit earlier

[cyan slate]

nice!

make a separate progress thread for uhda

[molten wyvern]

I decided to look into this a little more and it looks like it generates 3 spurious power button press querys + releases, for now what I did is just discarding all the events that are initially queued before enabling the gpe

linux also does that for some hardware but I didn't see my laptop in that list (though its funky in other ways like sometimes when you power it up the screen stays off for a couple seconds, then it resets and finally actually starts booting)

[molten wyvern]

a random progress image that doesn't really tell anything I have been making an api for vmx in the past few days and now I am trying to make seabios work in a "vm" using it

the (kernel) api itself is this ```cpp
int sys_evm_create(CrescentHandle& handle);
int sys_evm_create_vcpu(CrescentHandle handle, CrescentHandle& vcpu_handle, EvmGuestState** guest_state);
int sys_evm_map(CrescentHandle handle, size_t guest, void* host, size_t size);
int sys_evm_unmap(CrescentHandle handle, size_t guest, size_t size);
int sys_evm_vcpu_run(CrescentHandle handle);
int sys_evm_vcpu_write_state(CrescentHandle handle, int changed_state);
int sys_evm_vcpu_read_state(CrescentHandle handle, int wanted_state);

I should also commit all the libc + proper llvm cross toolchain stuff that I have locally but I'd first like to switch to a proper meta build system instead of cmake externalproject

[molten wyvern]

I really hate meta build systems and I hate having to have a cross llvm to build the stuff too, it doesn't even take that long (its like 15min) but I just hate the idea of it

and I think the input lag is more worse yet again

[dire galleon]

Very nice

I was thinking about having a go at hypervisor stuff myself in the future

Are you thinking of supporting svm too?

[molten wyvern]

likely, though I don't really have any real cpus to test it on myself

[dire galleon]

I have an amd system, I don't mind running some test images

[molten wyvern]

nice, I might take a look at it after I have figured all this other stuff out

and even for vmx there's a decent amount of stuff to do like emulating some of the instructions + some kind of api for injecting interrupts to the guest

[cyan slate]

damn thats pretty interesting

uKVM?

[molten wyvern]

lol

[cyan slate]

whats the current state of uhda btw?

[molten wyvern]

it works on all the hw I have tested it on but astral still has the weird bug even in qemu

and ofc it still lacks some things like the recording part

[cyan slate]

is it basically 1.0 now?

[molten wyvern]

eh, it needs more testing at least and I am not still not completely sure whether the api is good as I find it weird that there can be output groups with billion "logical outputs" that all go to the same physical speaker yet there can also be groups of individual outputs that go to different physical jacks (but idk how to make it better either, its the best so far and I'd have to first figure out how you are supposed to expose them before I could implement a better api for it)

[scenic rampart]

is it astral or uhda relate

d

[molten wyvern]

idk

the exact same write loop worked just fine in my kernel but ofc its not the only thing that there is different

and considering the "old" astral hda driver works just fine I think its something with the way I integrated the new one

[cyan slate]

astral doesnt have bugs thats a known fact

[scenic rampart]

is it pushed on the fork I can take a peek at it

[molten wyvern]

I can push it to a different branch

[scenic rampart]

sure

what exactly is the bug

[molten wyvern]

I think I posted a clip of it at some point, basically there is some random background noise

https://github.com/Qwinci/Astral/tree/uhda

[scenic rampart]

do you assume physical memory allocated is zero'd?
also: is size always guaranteed to be 0x1000 in physical allocation? otherwise that / 0x1000 would break (and its not portable :p)

[molten wyvern]

yes its always 0x1000 and I don't think I zero out the memory in my kernel either

[cyan slate]

well qemu anon memory is zeroed by default

[scenic rampart]

hmmmm

yeah by the point you do the cat into dsp quite a few programs ran already

well it did work on real hardware so

[cyan slate]

why is it crackling like that also?

[scenic rampart]

I think thats the bug qwinci is talking ab0ut

[molten wyvern]

yes

[cyan slate]

ah i thought he was talking about the pop at the beginning

[scenic rampart]

I think the pop is just header data lmao

[cyan slate]

can u just dump raw wav data somehow?

[molten wyvern]

?

[cyan slate]

so u skip the header and dont cause a pop

[molten wyvern]

yeah the proper way would be to obviously to read the headers and then only play the actual data lol

but for testing playback the headers don't really matter

[cyan slate]

i see

[scenic rampart]

yeah I have no idea, maybe add a memset(0) from the hddm on the allocated physical pages

because my only guess for that weird noise would be unitialized data

[molten wyvern]

also related to this I am not sure what should I do about the fact that I don't want to build llvm every time on the ci, it would make single build take like 3h on the garbage that the ci runs on

I'd just upload an archive with it and then download it but idk where, github doesn't allow that big files

unless I split it to multiple parts and then join but that's kinda cursed

ig Ill do that, it accepted 6 90mb files to the repo (though with a warning)

[molten wyvern]

it works its not even that much slower than the old ci, like 1min 20s for x86_64 and then probably around the same time for aarch64 (for reference previously it was 1min for both)

aarch64 is broken tho, Ill have to fix that

and update build guide in the readme

[molten wyvern]

idk if I really like these changes tho, I liked the old build system where you could build everything + run qemu using one cmake target, now you have to do ```bash
make # or
qpkg rebuild crescent crescent-apps # or for aarch64
make ARCH=aarch64 # or
qpkg --config=qpkg_aarch64.toml rebuild crescent crescent-apps

ran once

make create_image

make update_image run

[molten wyvern]

SeaBIOS (version rel-1.16.3-24-g62a1429e-dirty-20241127_163149-gentoo)
BUILD: gcc: (Gentoo 14.2.1_p20241026 p3) 14.2.1 20241026 binutils: (Gentoo 2.42 p6) 2.42.0
No Xen hypervisor found.
enabling shadow ram
Running on QEMU (q35)
physbits: signature="GenuineIntel", pae=yes, lm=yes, phys-bits=46, valid=yes
cpuid 0x40000000: eax 40000001, signature 'crescentevm'
RamSize: 0x05000000 [cmos]
Relocating init from 0x000d1ce0 to 0x03fea7c0 (size 87968)
=== PCI bus & bridge init ===
PCI: pci_bios_init_bus_rec bus = 0x0
=== PCI device probing ===
Found 1 PCI devices (max PCI bus is 00)
PCIe: using q35 mmconfig at 0xb0000000
=== PCI new allocation pass #1 ===
PCI: check devices
=== PCI new allocation pass #2 ===
PCI: IO: c000 - bfff
PCI: 32: 00000000c0000000 - 00000000fec00000
PCI: init bdf=00:00.0 id=8086:29c0
PCI: No VGA devices found
rdmsr is a stub
Found 1 cpu(s) max supported 1 cpu(s)
Copying PIR from 0x03fffc20 to 0x000f4900
Copying MPTABLE from 0x00006d84/3fe26b0 to 0x000f4820
Copying SMBIOS from 0x00006d84 to 0x000f46f0
load ACPI tables
init ACPI tables

now seabios gets past the qemu platform init but it wants pit and idk how do I emulate that in a good way (and it uses it to figure out tsc speed)

[drowsy belfry]

what you doing?

[cyan slate]

uKVM

[cyan slate]

thats quite a lot already tbh

[hot cobalt]

this will be very cool

though i wonder how the kernel interface will look

[cyan slate]

thats an interesting question

[drowsy belfry]

is that for real

[molten wyvern]

making a hypervisor (vmx only for now) + some sort of toy vm

[cyan slate]

is it internal to your kernel or will it actually be like uKVM

[molten wyvern]

well it could def be made like that as it doesn't really depend on anything other than a function to allocate singular physical pages + some sort of way to map them

but rn its not like that, its just inside the kernel

[cyan slate]

ukvm_kernel_handle_ioctl

then u can literally run QEMU with kvm

or rather ukvm_kernel_install_ioctl_handler

i'd have to do a similar thing for uGPU

[molten wyvern]

the actual kvm api also has things like in-kernel io-apic and pit emulation stuff

[cyan slate]

well there are kvm capabilities and stuff

u can just tell the process its unsupported

[molten wyvern]

yeah maybe its not necessary

[cyan slate]

qemu has fallback software emulation for all of that stuff

[molten wyvern]

one slightly annoying thing that I haven't figured out is moving vcpu threads across cores because you don't necessarily know ahead of time that the thread is going to be moved if there is a separate load balancer on a different core

as you'd have to execute vmclear on the old core before starting to execute it on the new core

as a poor workaround ig you could always unconditionally run vmclear after every vmexit but then you can't use vmresume which likely makes performance worse

[cyan slate]

yeah you'd need some migration hook

[molten wyvern]

thread migration thread on every core and then when the load balancer wants to move a thread it just inserts the (thread, new cpu) pair to a workqueue on the old cpu's thread migration thread and then when that thread runs it can call any hooks that there might be before moving it

actually ig that's not the worst solution ever as long as its high priority, its not that thread migrations are that important so they'd have to happen instantly

or alternatively a different load balancing design where cpus would push threads to other cpus themself

[cyan slate]

usually for high perf vms u pin the vcpu thread to a specific cpu tbh

[molten wyvern]

yeah that's what I do now

[cyan slate]

and ideally remove that cpu from the scheduler pool completely

isolcpus=... on linux

[molten wyvern]

I got an idea for the pit too, maybe if I first fix the tsc so only the time actually spent in the vm is visible then I can just use whatever the tsc frequency is to calculate the amount of pit ticks that happened at the time between the last vmenter and vmexit

it does mean that it might jump quite a bit in time without the cpu seeing the intermediate states but idk if it would really matter

[molten wyvern]

this worked fine ^ now it gets to a vga option rom that I added, it dies there tho ```
Checking rom 0x000c0000 (sig aa55 size 76)
Running option rom at c000:0003
Start SeaVGABIOS (version rel-1.16.3-24-g62a1429e-dirty-20241128_163504-gentoo)
VGABUILD: gcc: (Gentoo 14.2.1_p20241026 p3) 14.2.1 20241026 binutils: (Gentoo 2.42 p6) 2.42.0
enter vga_post:
a=00000008 b=0000ffff c=00000000 d=0000ffff ds=0000 es=f000 ss=0000
si=00000000 di=000051c0 bp=00000000 sp=00006cd6 cs=f000 ip=c999 f=0000
No VBE DISPI interface detected, falling back to stdvga
Attempting to allocate 512 bytes lowmem via pmm call to f000:ca50
pmm call arg1=0
pmm00: length=20 handle=ffffffff flags=9
VGA stack allocated at e5de0
guest ip 6CB053EB
KERNEL PANIC: vmx.cpp:933: (operator()) assertion 'host' failed

it somehow gets to that (seemingly garbage) 0x6cb053eb ip and then the kernel dies because it can't decode the instruction there (ik it shouldn't really die but for now™️ its fine)

[molten wyvern]

now it works 💀 I didn't even change anything (except recompiled seavgabios as I were trying to add a debug print but even without that print it works)

[molten wyvern]

there are a ton of unhandled vga port writes tho (along with other unimplemented things like ps2 + pit irqs + rtc stuff) and I hardcoded the width/height of the font that the bios writes to the vram

[lament drift]

oooh that's sick!!

nice

[molten wyvern]

yeah its pretty cool even tho I kinda hacked the vga printing together and it just prints it once at the end

I guess Ill have to have some sort of timeout in both the vm run loop and the kernel side of it so the thread doesn't get stuck in the vm forever if it has a while (true) there without any instructions that would cause vm exits (or well it doesn't really get stuck right now either, it gets pre-empted but it doesn't ever return back to the userspace as it enters the vm again after it resumes)

then I can draw every time after that run loop or whatever

[lament drift]

idk how linux handles that

if it even does

or is that on the e.g qemu side

[cyan slate]

it's on the qemu side of course

the vmexit is handled in userspace

[dire galleon]

thats cool! Keen to read through your implementation at some point 😄

[molten wyvern]

I took a look at what would it take to port openjdk to a completely new platform like my os and oh god its a lot of things 💀 I mean its not impossible but I don't really want to do that either so I am trying to think what should I do, realistically if I want to port anything bigger like that in somewhat reasonable time I need either some kind of posix/linux compat or nt compat

[rocky echo]

why don't you just switch to linux abi

it makes life so much easier

[cyan slate]

nt compat is more based

[molten wyvern]

the thing about that is that I'd like to do it fully if going that route (as in having the kernel be a PE, having drivers work with the kernel api and you can use other fun stuff like __throw/__except too) but there are also some slightly annoying things with that like gdb kinda sucks for windows targets (you can step in to functions but the backtrace gets fed and you can't step out without it losing control)

well ig I could also do only like userspace compat but then it would need me to write opengl or whatever drivers which idk if I really want to do at least not rn (or well idk how hard would mesa be to port to a weird mixed platform like that, maybe it would be an option too)

or for the option of posix/linux compat one idea I had was to do basically something similar to what microkernels do and have a posix server and as minimal posix stuff in the kernel as possible (though that would probably be really bad for performance with my ipc socket impl )

or just otherwise abstracted inside the kernel, idk

[cyan slate]

implement windbg compat, ez

[molten wyvern]

then I'd have to run that in a vm too lol

and idk what it needs

[cyan slate]

wdym?

windbg works over fake serial

[molten wyvern]

yeah but I can't run windbg on linux

[cyan slate]

surely wine can run it lol

[molten wyvern]

well maybe

[cyan slate]

also, an nt-like kernel must be developed on NT

[molten wyvern]

anyway ig Ill try to make it now that I already started, just have to hope that I don't get too many hard to debug bugs

[cyan slate]

cant you compile it as both elf and pe

then feed elf to gdb

and run pe

[molten wyvern]

idk if the windows target supports outputting elfs, I have doubts about that

[cyan slate]

hmm

maybe

[molten wyvern]

actually maybe it does

ig I can try what gdb thinks about that

[scenic rampart]

sorry wrong chat

[ivory cloak]

smh

[molten wyvern]

yeah no, I just realized that while clang can output elf files with that target when I link them the resulting addresses are going to be different

[cyan slate]

isnt there a way to tell gdb the base

[molten wyvern]

yeah but like the sections likely end up being different in relation to each other

I also though lldb would work better with pe's but I couldn't get it to resolve any symbols or sources at all

[cyan slate]

hmm maybe

or make a pe->elf converter

if one doesnt exist already

[molten wyvern]

idk if that makes it any better tho, it might just be that it doesn't like something about the call conv or the debug info

but eh ig its not too bad, at least it somewhat works

[molten wyvern]

I finally pushed the vmx stuff, I had that locally for like a month (and the userspace vm thing is still very incomplete as it was a month ago)

[molten wyvern]

also I think I am going to do some posix/linux stuff after all because I don't really feel like writing a completely new kernel rn

[molten wyvern]

I am running into qpkg skill issues again or well its not that bad but when I rebuild a package using it it somehow causes literally all files to get rebuilt instead of only the changes ones (it doesn't delete the build dir or anything either)

[molten wyvern]

actually I think I know why, the libc includes its own headers from the sysroot and they get touched when installing

[karmic coral]

@molten wyvern

[molten wyvern]

does anything happen if you close the lid?

[karmic coral]

nop

or well, i tried the brightness keys but those also go through the ec

or do you specifically install the notify handler for the lid

[molten wyvern]

no, it should print for any ec event

I guess my ec impl is broken somehow then too

[karmic coral]

also how did you find a ps2 mouse

[clever shell]

hi qwinci

i love crescent

because im a crescent

[karmic coral]

this laptop only has a ps/2 keyboard

[clever shell]

https://tenor.com/view/doge-shiba-inu-dog-croissant-gif-18352361

[karmic coral]

the touchpad is hid over i2c + gpio for irqs

[molten wyvern]

hmm

yeah I have no idea about that because that means it generated some response byte

[cyan slate]

Bruh everyone is having skill issues with ec

[molten wyvern]

somehow on the two laptops that I tested qacpi on the ec happened to work fine

I should also add some more prints, rn I literally don't print anything when an ec is found

[molten wyvern]

wait did it hang here?

I just realized, that's not where I put the loop at

[karmic coral]

yes nothing printed afterwards

[molten wyvern]

I guess that might explain it then, it got stuck somewhere in the event init code with irqs disabled

[scenic rampart]

just dont bother with it

the heart wont feel what the eyes cant see

[molten wyvern]

@karmic coral what does the ec node's _CRS contain?

because the only sensible places where it could get stuck is crs parsing or clearing ec events at the start

[cyan slate]

Btw are u sure u aren't truncating io addresses to a u8

I had this bug when working on managarm ec

[karmic coral]

            Method (_CRS, 0, NotSerialized)  // _CRS: Current Resource Settings
            {
                Name (BFFR, ResourceTemplate ()
                {
                    IO (Decode16,
                        0x0062,             // Range Minimum
                        0x0062,             // Range Maximum
                        0x00,               // Alignment
                        0x01,               // Length
                        )
                    IO (Decode16,
                        0x0066,             // Range Minimum
                        0x0066,             // Range Maximum
                        0x00,               // Alignment
                        0x01,               // Length
                        )
                })
                Return (BFFR) /* \_SB_.PCI0.SBRG.EC0_._CRS.BFFR */
            }

[cyan slate]

Nvm this one is rather trivial

One of my laptops has ec like somewhere very high in io space

[molten wyvern]

yeah I don't think that should cause any issues, its just normal io descriptors

here is an iso with more prints

[karmic coral]

lemme test

[cyan slate]

BTW even to this day I dont understand why there's a min and max range field

Why do that when there's a length field anyway

[karmic coral]

multiple length-sized areas within the min/max range?

[cyan slate]

Perhaps, but what would you do if min ! = max

[molten wyvern]

so it doesn't like this ```cpp
for (int i = 0; i < 100; ++i) {
enable_burst();
write_control(ec_cmd::QR_EC);
u8 code = read_data();
disable_burst();
if (!code) {
break;
}
else if (i == 99) {
println("[kernel][acpi]: warning: ec keeps generating events");
}

println("[kernel][acpi]: discarding ec query ", Fmt::Hex, code, Fmt::Reset);

}

ig I could make waiting for the status bits have a timeout (though to be fair this whole code shouldn't be needed on most EC's)

[cyan slate]

Fwiw Linux doesn't have a timeout quirk for any ecs

Although it still does handle hangs

[molten wyvern]

linux also doesn't unconditionally do this kind of thing

at least to my understanding when I last looked

[cyan slate]

Which thing?

[molten wyvern]

clearing ec events at the start when it finds an ec

[cyan slate]

That's true

[molten wyvern]

I just did that to get rid of the spurious power button events I got on my laptop sometimes and its not even on the linux quirk list

[cyan slate]

Still waiting for u to test some uacpi os to see if it gets those or not

[molten wyvern]

ig I can flash managarm to an usb and test it rn when I remember it

[cyan slate]

That'd be cool if u got an image handy

[molten wyvern]

doesn't look like it does at least in like 10 tries or whatever I tried

maybe its a qacpi skill issue

[cyan slate]

Any chance qacpi doesn't handle some field io correctly

Like indexfield perhaps