OBOS (not vibecoded) | Operating System Development | Page 12

flint idol Jul 31, 2024, 11:12 AM

#

by changing the code at pg_cmp_pages

#

I figured that the compiler was optimizing this:

if (left->addr == right->addr)
  return 0;
return (left->addr < right->addr) ? -1 : 1;```

#

into something to the likes of this:

return (intptr_t)left->addr - (intptr_t)right->addr;

#

which was causing problems for me previously

#

so I changed to debug mode and uncommented that line and was able to reproduce the bug

#

now to fix it

flint idol Jul 31, 2024, 11:41 AM

#

gcc be doing some stupid shit

#

I want to insert a break point like this:

volatile bool b = true;
while (b)
  b = true;

#

and gcc optimizes it to while(true)

flint idol Jul 31, 2024, 11:45 AM

#

flint idol into something to the likes of this: ```c return (intptr_t)left->addr - (intptr_...

except I'm not quite sure how fucking subtraction can fail

#

why is the rb tree implementation passing in the same page node to pg_cmp_pages each time

#

what is this shit

#

TODO: Fix these stupid bugs with optimizations

#

I cannot be bothered this early to fix the compiler being stupid

#

I haven't changed shit since yesterday

#

why

#

the

#

fuck

#

did it stop working

#

[ ERROR ] Could not load test driver #2. Status: 11.

#

wtf is this

#

all I changed was the section the driver header was in

#

(note: the section works fine for the first test driver)

#

probably the compiler doing weird stuff

#

fixed it

#

since I couldn't be bothered to fix those other bugs atm

#

I'm going to start with the driver interface part of the driver interface

#

this will be dictated by a flag in the driver header

flint idol Jul 31, 2024, 12:29 PM

#

Note: there will be more callbacks than just read/write

#

So far, I have 11 callbacks

flint idol Jul 31, 2024, 12:48 PM

#

I think this looks good so far:

typedef struct driver_ftable
{
    // Note: If there is not an OBOS_STATUS for an error that a driver needs to return, rather choose the error closest to the error that you want to report,
    // or return OBOS_STATUS_INTERNAL_ERROR.

    // ---------------------------------------
    // ------- START GENERIC FUNCTIONS -------
    
    obos_status(*get_blk_size)(dev_desc desc, size_t* blkSize);
    obos_status(*get_max_blk_count)(dev_desc desc, size_t* count);
    obos_status(*read_sync)(dev_desc desc, void* buf, size_t blkCount);
    obos_status(*write_sync)(dev_desc desc, void* buf, size_t blkCount);
    obos_status(*foreach_device)(iterate_decision(*cb)(dev_desc desc, size_t blkSize, size_t blkCount));
    // The driver dictates what the request means, and what its parameters are.
    obos_status(*ioctl)(size_t nParameters, uint64_t request, ...);
    
    // -------- END GENERIC FUNCTIONS --------
    // ---------------------------------------
    // ---------- START FS FUNCTIONS ---------

    // NOTE: FS Drivers must always return one from get_blk_size
    // get_max_blk_count is the equivalent to get_filesize

    // lifetime of *path is dicated by the driver.
    obos_status(*query_path)(dev_desc desc, const char** path);
    obos_status(*path_search)(dev_desc* found, const char* what);
    obos_status(*move_desc_to)(dev_desc desc, const char* where);
    obos_status(*mk_file)(dev_desc* newDesc, dev_desc parent, const char* name, file_type type);
    obos_status(*remove_file)(dev_desc desc);
    obos_status(*get_file_perms)(dev_desc desc, file_perm *perm);
    obos_status(*set_file_perms)(dev_desc desc, file_perm newperm);
    obos_status(*list_dir)(dev_desc dir, iterate_decision(*cb)(dev_desc desc, size_t blkSize, size_t blkCount));
    // ----------- END FS FUNCTIONS ----------
    // ---------------------------------------
} driver_ftable;```

#

I think any devices that like communicate

#

can just use an ioctl to initialize the connection

#

I almost forgot to add a callback to query a "file" type

#

and by file I mean directory entry

#

file_perm is simply just unix perms

#

typedef struct file_perm
{
    bool other_exec : 1;
    bool other_write : 1;
    bool other_read : 1;
    bool group_exec : 1;
    bool group_write : 1;
    bool group_read : 1;
    bool owner_exec : 1;
    bool owner_write : 1;
    bool owner_read : 1;
} OBOS_PACK file_perm;```

#

by driver interface of the driver interface, I basically just meant that for now

#

since I have nothing else to expose to drivers

inland radish Jul 31, 2024, 1:00 PM

#

flint idol I almost forgot to add a callback to query a "file" type

oh, like stat I guess?

flint idol Jul 31, 2024, 1:00 PM

#

soon enough, I will be exposing ways to access disks, ways to access the PCI bus, etc.

flint idol Jul 31, 2024, 1:00 PM

#

inland radish oh, like stat I guess?

kinda, except not stat

inland radish Jul 31, 2024, 1:00 PM

#

flint idol I think this looks good so far: ``` typedef struct driver_ftable { // Note: ...

these operations look more like what you'd apply on a vnode

#

like vnode->read_sync

flint idol Jul 31, 2024, 1:01 PM

#

exactly

inland radish Jul 31, 2024, 1:01 PM

#

so these are intended for vnodes right

inland radish Jul 31, 2024, 1:01 PM

#

flint idol soon enough, I will be exposing ways to access disks, ways to access the PCI bus...

who are you exposing these to

flint idol Jul 31, 2024, 1:01 PM

#

drivers

inland radish Jul 31, 2024, 1:01 PM

#

mhm

#

pretty cool

#

i'm now finishing up my work and committing it

flint idol Jul 31, 2024, 1:02 PM

#

inland radish so these are intended for vnodes right

the first 6 functions are just for any driver that has rw capability to expose

#

that function table is incomplete though

#

I'm still changing stuff

#

you just reminded me to change read_sync/write_sync to also take in a block offset

#

which does make it incompatible with any sort of communication driver

#

and I don't want to do something stupid

inland radish Jul 31, 2024, 1:05 PM

#

flint idol which does make it incompatible with any sort of communication driver

no

#

the offset will simply be ignored for pipe style devices

flint idol Jul 31, 2024, 1:05 PM

#

oh yeah

#

I'll just add a flag in the driver header that dictates whether the device is a pipe-style device or some other device

#

https://github.com/OBOS-dev/obos/blob/6e039ae43e1e0e3793b39bc9c5fa41e8636f5dd1/src/oboskrnl/driver_interface/header.h

GitHub

obos/src/oboskrnl/driver_interface/header.h at 6e039ae43e1e0e3793b3...

I was bored... Contribute to OBOS-dev/obos development by creating an account on GitHub.

#

there's the driver header + function table if anyone was curious

#

since serial drivers are so damn simple, I think I'll write one

#

(for testing)

#

I need to implement unloading drivers

#

# Check driver->refCnt
If driver->refCnt == 1:
  continue
else:
  return STATUS_IN_USE
driver->DriverCleanupCallback()
ForEach Dependency = Dependency:
  Dependency->refCnt--
RemoveDriverFromList(Drv_LoadedDriver, driver)
ForEach Page = MappedPageInDriver:
  VirtualMemoryFree(page)
FreeDriverId(driver)```

#

I think that's it for unloading drivers

flint idol Jul 31, 2024, 1:43 PM

#

I'm done that

#

now to test it

flint idol Jul 31, 2024, 1:59 PM

#

after some changes

#

it almost works (tm)

#

the scheduler deadlocked

#

while exitting a thread

#

seemed to be something rare

#

as I was unable to reproduce it on the next boot

#

It works

#

switching the order of the unloads breaks (intended, as driver 2 depends on driver 1, but wasn't unloaded yet)

inland radish Jul 31, 2024, 2:06 PM

#

flint idol while exitting a thread

https://github.com/OBOS-dev/obos/blob/528165279892ffaafe70b6b1092ff58e5c4ce381/src/oboskrnl/scheduler/thread.c#L194 you probably shouldnt do this

flint idol Jul 31, 2024, 2:07 PM

#

look one line above

#

that's a use after free in the scheduler

#

if I don't

inland radish Jul 31, 2024, 2:08 PM

#

then dont free the thread there

#

free it in some kind of reaper or something

#

or a DPC

flint idol Jul 31, 2024, 2:10 PM

#

I don't really see the point in that if the thread can be freed immediately

#

(the scheduler doesn't rely on the current thread to schedule)

#

I think I might know why the scheduler occasionally deadlocks

#

it is possible that something takes the global scheduler lock after the per-cpu scheduler lock

#

but then something in the scheduler takes the per-cpu lock of the cpu that was just locked

#

but then since that thread is waiting for the global scheduler lock

#

those two cpus deadlock

#

causing other cpus waiting for the global scheduler lock to deadlock

#

to test that theory out, I will set OBOS to compile as Uniprocessor

#

and test on real hw where this issue prevails

#

but before I do that, I'm going to make the ready thread function lock the scheduler

#

as apparently it wasn't doing that before

#

I'm done that

#

I will now reboot into OBOS

#

the Uniprocessor build PFs on qemu

#

well that's weird

#

the page appears to be paged out

#

not this again

inland radish Jul 31, 2024, 2:43 PM

#

flint idol I don't really see the point in that if the thread can be freed immediately

freeing really should be deferred to where the thread is completely out of the scheduler system

flint idol Jul 31, 2024, 2:44 PM

#

oh yeah that's a good point

#

especially considering how slow the allocator can sometimes be

flint idol Jul 31, 2024, 3:02 PM

#

flint idol the page appears to be paged out

I can rule out it being a race condition

#

well...

#

MmS_SetPageMapping returned a bogus status

#

oh my days

#

the reason is quite funny

#

I forgot to return a value from invlpg_impl (used to send tlb shootdowns)

#

with OBOS compiled as uniprocessor, the tlb shootdown part is compiled out

#

and only the invlpg part is kept

#

so theoretically, in the MP build, it should've also failed

#

but it didn't because the last statement called a function which returned OBOS_STATUS_SUCCESS

#

and because gcc hadn't modfied rax after

#

it would be as if the function return success

#

tl;dr, I forgot to return a value from a function

#

after fixing that it boots

#

and I can now test obos on real hw with a UP build

#

Damn it it still hangs

ashen goblet Jul 31, 2024, 3:23 PM

#

Are you not using -Wreturn-type or whatever it’s called?

#

(It‘s part of -Wall)

flint idol Jul 31, 2024, 3:23 PM

#

I have -Wall and -Wextra

#

so yes

#

but I think I just ignore most warnings

ashen goblet Jul 31, 2024, 3:23 PM

#

and it doesn’t warn you about missing return values?

flint idol Jul 31, 2024, 3:23 PM

#

flint idol but I think I just ignore most warnings

.

ashen goblet Jul 31, 2024, 3:24 PM

#

flint idol but I think I just ignore most warnings

nooo

#

There‘s -Werror for a reason

flint idol Jul 31, 2024, 3:24 PM

#

I don't like that because of compatibility with future compiler versions

#

I'll see what happens if I disable the timer interface

#

since as we all know, disabling random kernel components always helps

ashen goblet Jul 31, 2024, 3:26 PM

#

flint idol I don't like that because of compatibility with future compiler versions

if you change compiler version you‘re going to have to change your toolchain build anyways it‘s not like it‘ll break by itself

flint idol Jul 31, 2024, 3:26 PM

#

I don't have a toolchain

#

I use a generic x86_64-elf compiler

#

Oh and btw it still hung

ashen goblet Jul 31, 2024, 3:27 PM

#

flint idol I don't have a toolchain

oh mb then

flint idol Jul 31, 2024, 3:27 PM

#

I will eventually

#

Just not now as there's no point (I don't want to port gcc)

ashen goblet Jul 31, 2024, 3:28 PM

#

atm obos only compiles for x86_64 anyways right

flint idol Jul 31, 2024, 3:28 PM

#

right

#

I'll eventually port to like

#

the m68k

#

or something like that

#

but for now just x86-64

#

I made the spinlock display a warning after it spins for too long

#

and print a stack trace

#

Of course nothing gets printed

#

As something continues to hang

#

It's been doing nothing for about 5 minutes now

#

@real pecan During namespace init, does uACPI start any defferred work using uacpi_kernel_schedule_work

#

well there goes infy lol

#

that could actually be the case

#

work was scheduled

#

then uacpi_kernel_wait_for_work_completion was called

#

but then that hangs

#

as for why it'd only hang now

#

idk

#

Well it doesn't hang there

#

If I were to commend out uacpi init

#

*comment

real pecan Jul 31, 2024, 3:52 PM

#

flint idol <@353593948523986945> During namespace init, does uACPI start any defferred work...

Possibly

real pecan Jul 31, 2024, 3:52 PM

#

flint idol then `uacpi_kernel_wait_for_work_completion` was called

Thats never called unless you delete gpe blocks yourself

flint idol Jul 31, 2024, 3:52 PM

#

ok then

#

I'll just see if this has anything to do with uACPI first

#

or if it's purely in the scheduler

flint idol Jul 31, 2024, 3:53 PM

#

flint idol I'll just see if this has anything to do with uACPI first

as in, faulty kernel api

#

It's somewhere in uacpi

#

Possibly within GPE execution or something similar

#

As when irqs are masked during uacpi init, it just hangs after init

#

But I can confirm the driver loader works on real hw

#

I just realized this is possibly before acpi mode is enabled

#

my logging is too slow I need to fix that eventually

#

I can fix it by adding a backbuffer or using flanterm

#

wait I thought I mapped the framebuffer as WT

#

it turned out it wasn't mapped as anything special at all

flint idol Jul 31, 2024, 5:45 PM

#

I did that

#

Omg it's so much faster

#

I don't even have double buffering

#

I think I'll take a bit to make that

#

It hangs after initializing a gpe block

ashen goblet Jul 31, 2024, 5:51 PM

#

flint idol it turned out it wasn't mapped as anything special at all

fun fact: on some hardware the perf difference will me massive while on other it will be merely noticeable

flint idol Jul 31, 2024, 5:53 PM

#

in my case it was infinitely faster

ashen goblet Jul 31, 2024, 5:54 PM

#

qemu or real hw?

flint idol Jul 31, 2024, 5:54 PM

#

real hw

#

qemu was barely noticeable

ashen goblet Jul 31, 2024, 5:54 PM

#

yeah same for me

flint idol Jul 31, 2024, 5:55 PM

#

maybe I should port flanterm

#

because of how much printf logging I do on real hw

#

meh too much work

#

@real pecan What does uACPI do after GPE initialization, as my kernel hangs right after that

#

it said something like GPE block initialization complete

#

then hung

flint idol Jul 31, 2024, 6:18 PM

#

the backbuffer nearly works

#

except after 32 lines it stops working

#

I fixed that

#

just for it to pf on access of the backbuffer later

#

it pfed in memcpy which was called from flush_buffers

#

from what I can tell, it copied 0x4000 bytes before crashing

#

now because I'm bigbrain, I made sure to disable the back buffer before panic

real pecan Jul 31, 2024, 6:33 PM

#

flint idol <@353593948523986945> What does uACPI do after GPE initialization, as my kernel ...

i mean a lot of things lol, look at the code

#

probably irq handler installation

flint idol Jul 31, 2024, 6:33 PM

#

well that could be it

#

uacpi_kernel_install_interrupt_handler?

real pecan Jul 31, 2024, 6:41 PM

#

Yes

flint idol Jul 31, 2024, 6:57 PM

#

I fixed the back buffer

flint idol Jul 31, 2024, 7:01 PM

#

flint idol `uacpi_kernel_install_interrupt_handler`?

I added some logs to this

#

time to see what it gives me

#

inb4 it hangs while unmasking the irq

#

DAMN IT

#

#

It indeed does

#

my theory is since the irq handler wasn't set in the object yet, the irq interface somehow just froze

#

after receiving a GPE

#

I'll swap those two and see what happens

#

I'm not sure whether to be happy it still freezes

#

Or not

#

I added some logs to the IRQ dispatcher

#

to see whether it hangs in there

#

inb4 it hangs while acquiring the lock

#

WHY THE HELL ARE THERE SO MANY GPES

#

It's just infinite gpe recursion

#

It just logs that it's in the irq dispatcher

#

Then says it's trying to take the log

#

Thsn it's just infinite recursion

#

I think I might know why

#

I use some random settings for the GSI 9

#

like polarity and stuff

#

because I assumed there would be a MADT redirection entry for it

#

and iirc there is on my computer which I'm testing on

#

there indeed is

#

it's active low and set to trigger mode sensitive

#

which are the polar opposites of what I pass in

#

but the ioapic code should've find a redirection entry for it

#

and set the trigger mode and polarity accordingly

#

qemu has the same redirection entry

#

and works fine

#

could be a bug with MADT init code

#

I'll reboot and find out

#

Yeah it does recognize that

flint idol Jul 31, 2024, 7:59 PM

#

So it turned out I wasn't supposed to unmask the GSIs right away

#

now I just have to find out why the GPE isn't running

#

(yes I do unmask it)

real pecan Jul 31, 2024, 8:01 PM

#

flint idol qemu has the same redirection entry

Qemu probably doesn't care

flint idol Jul 31, 2024, 8:06 PM

#

flint idol So it turned out I wasn't supposed to unmask the GSIs right away

.

flint idol Jul 31, 2024, 8:06 PM

#

real pecan Qemu probably doesn't care

but yeah probably

real pecan Jul 31, 2024, 8:17 PM

#

https://github.com/UltraOS/uACPI/blob/master/source/event.c#L188

#

status_mask and enable_mask here

flint idol Jul 31, 2024, 8:17 PM

#

wrong channel but ok

real pecan Jul 31, 2024, 8:18 PM

#

https://github.com/UltraOS/uACPI/blob/d096e2b3d708428ac76daa9fecb90052c0b380b2/source/event.c#L539

#

status here

real pecan Jul 31, 2024, 8:18 PM

#

flint idol wrong channel but ok

i mean we're debugging your os caret

flint idol Jul 31, 2024, 8:18 PM

#

fair fair

#

I'm getting infinite 0x0 from uACPI's gpe handler

#

Which I assume means unhandled

real pecan Jul 31, 2024, 8:22 PM

#

did u log both status masks?

flint idol Jul 31, 2024, 8:22 PM

#

No I just logged the final return of the handler passed in install interrupt handler

real pecan Jul 31, 2024, 8:22 PM

#

that doesnt necessarily mean anything

#

log the masks

#

they come from hardware

flint idol Jul 31, 2024, 8:23 PM

#

K

#

Bruh

#

It hangs

#

When I added the logs

real pecan Jul 31, 2024, 8:26 PM

#

lol

flint idol Jul 31, 2024, 8:26 PM

#

Probably got the gpe while the logger was locked

#

So it decided to deadlocj

#

So to solve that, I will just make it panic, since that unlocks the logger

real pecan Jul 31, 2024, 8:27 PM

#

that would be the first log msg tho

#

which might have set masks

flint idol Jul 31, 2024, 8:28 PM

#

Yeah nvm

#

I think those lines just aren't ever hit

#

btw it hangs right after "initialized GPE block, ... (IRQ9)"

real pecan Jul 31, 2024, 9:27 PM

#

Yeah no idea, your kernel isnt even invoking the handler then?

flint idol Jul 31, 2024, 9:31 PM

#

no it is

#

y'know what I'll just keep the irq masked until I can figure out the problem

#

as now I want to write a simple serial driver

flint idol Jul 31, 2024, 10:56 PM

#

I think I'm going to delete the test drivers

#

as they're now useless

flint idol Jul 31, 2024, 11:14 PM

#

I used uACPI to detect the avaliable COM ports on the system:

[ DEBUG ] COM1 is from IO address 0x3f8 - 0x400, and is on GSI 4.```

flint idol Jul 31, 2024, 11:34 PM

#

It's so peaceful coding a driver

#

In a nice environment

#

where you don't need to making everything

flint idol Jul 31, 2024, 11:35 PM

#

flint idol It's so peaceful coding a driver

*com port driver

#

otherwise everything I just said has been rendered void

flint idol Aug 1, 2024, 1:30 AM

#

the base of the uart driver is done

#

it can receive

#

data

#

I just need to see if it can send data

#

then I need to expose those functions

#

tomorrow I'll be working on PnP

#

then I'll work on DPCs

#

since I don't have those yet

#

then I can merge the driver interface

#

took a lot less time than I expected

#

I'll also implement an initrd of sorts

#

I think I'll done the driver interface in ~3 days

#

depends if I work on it or not

flint idol Aug 1, 2024, 2:04 AM

#

after the driver interface, I'm making a gdb stub

#

well writing to the serial port doesn't quite work lol

#

Assertion failed in function pop_from_buffer. File: /home/oberrow/Code/obos/src/drivers/x86_64/uart/serial_port.c, line 69. buf->buf

#

I fixed that by removing the assert

#

since it wasn't supposed to be there

flint idol Aug 1, 2024, 2:20 AM

#

flint idol

I was reading from the struct buf instead of the pointer to the buffer

#

in pop_from_buffer

#

thus causing it to give that random shit

#

but I can now get a message

flint idol Aug 1, 2024, 2:54 AM

#

the buffering is kind of messed up

#

so I'll fix that tomorrow

#

flint idol Aug 1, 2024, 10:57 AM

#

I think a possible cause of the bug is lost IRQs

#

I think that was it

#

maybe I shouldn't be buffering writes within the driver

#

and let the kernel do that

flint idol Aug 1, 2024, 11:37 AM

#

flint idol

so I have been able to fix the bugs with the COM driver as seen here

#

I also implemented every required function in the function table

#

time to test to see if it works from within the kernel

#

ye it works

#

what if I were to just do the equivalent of dding a bunch of data on the serial port

flint idol Aug 1, 2024, 12:08 PM

#

just pushed that

flint idol Aug 1, 2024, 12:08 PM

#

flint idol tomorrow I'll be working on PnP

it's PnP time

#

wait that kinda needs a driver list

#

I'll just make one up

#

until I get an initrd

flint idol Aug 1, 2024, 12:25 PM

#

Fun fact: if you search the source code of OBOS, you can find me writing comments that look like:
// NOTE(oberrow, 23:47 2024-07-14):
where I basically record my thoughts

#

in the comments

#

.

#

(for reference while I make this)

#

time to implement a hashmap

#

https://github.com/tidwall/hashmap.c

#

this is a lot more convinient

flint idol Aug 1, 2024, 4:13 PM

#

I made the APIs to enumerate the pci bus

#

hopefully it's cross-platform enough

#

I can now start the pnp thing

#

nah I think it'll just be a config option to enable/disable it at boot

flint idol Aug 1, 2024, 10:08 PM

#

after 6 hours of nothing

#

I am going to continue working on this

flint idol Aug 1, 2024, 10:52 PM

#

I am done the part that divides the drivers into pci drivers and acpi drivers

flint idol Aug 1, 2024, 11:10 PM

#

the pci part should be done

#

it's like dead simple

#

searches the hashmap for the class code and stuff

#

for each driver

#

that supports the pci device

#

it adds the driver to the list of drivers to load

#

However, I do need to quickly implement vendor IDs here

#

it just checks the class code and stuff

#

not the vendor id though

#

it now checks the vendor/device id for each driver

#

assuming the driver says that the kernel should check for that

#

(it's a flag)

flint idol Aug 1, 2024, 11:39 PM

#

ok ACPI devices

#

kinda makes me wish uACPI had documentation

real pecan Aug 1, 2024, 11:46 PM

#

Header files + wiki

#

What else do u want documented

flint idol Aug 1, 2024, 11:46 PM

#

too much work to search header files

real pecan Aug 1, 2024, 11:46 PM

#

Lol

flint idol Aug 1, 2024, 11:46 PM

#

but I found what I needed on the osdev wiki

real pecan Aug 1, 2024, 11:47 PM

#

Nice

flint idol Aug 1, 2024, 11:48 PM

#

this pnp stuff was so messy in my third kernel

#

this can do like the exact same as the third kernel

#

but cleaner

#

(as in, looks cleaner)

#

the acpi thing is done

real pecan Aug 2, 2024, 12:08 AM

#

Big

flint idol Aug 2, 2024, 12:08 AM

#

now I'm implementing an api to get a driver header from the driver's binary

#

so that I can actually use the pnp thing

flint idol Aug 2, 2024, 12:11 AM

#

flint idol this is a lot more convinient

damn it

#

it uses floating points

#

I'll just use the fixed point library

#

it only uses floating points in one place

#

I fixed that

#

it compiles

#

time to test

#

but before that I need to check if the changes I made to the driver loader work

#

it not crashing probably means it works

#

the driver works

#

so I'll just assume that all works

#

unaligned access

#

lets go

#

fixed that

#

and execution of nullptr???

#

I wasn't compiling the pci functions

#

but I set them to weak

#

in case some arch didn't support them

#

so it was trying to call that

lean glen Aug 2, 2024, 12:39 AM

#

real pecan Header files + wiki

doxygen when?

flint idol Aug 2, 2024, 1:04 AM

#

UBSan Violation unaligned access in file /home/oberrow/Code/obos/src/oboskrnl/allocators/basic_allocator.c at line 406:7```

#

nooo

#

oopsie

#

I wasn't supposed to free elements from the hashmap

#

ASAN Violation at ffffffff8005b4f9 while trying to read 8 bytes from 0xaaaaa9aaaaf6115a.```

#

use after free

#

for some reason the hashmap is having trouble freeing itself

#

ok I fixed it

#

it successfully identifies the existence of the UART

#

and throws the UART driver into the list of drivers to load

#

now since the function only takes in driver headers, it doesn't load anything

#

but it spits out a list of drivers to load

flint idol Aug 2, 2024, 1:17 AM

#

flint idol it successfully identifies the existence of the UART

damn it

#

it stopped as soon as I said that

#

if (shouldAdd)
    goto end;```

#

I fixed that

#

in exactly 333 lines:

#

pnp

#

DPCs are next

#

just I'm unsure when to make them run though

#

actually

#

I think they run whenever IRQL is set to anything < DPC_LEVEL

#

just pushed everything

#

Current roadmap:
DPCs->Merge driver interface->GDB Stub->VFS->Syscalls->GCC Target->mlibc->a bunch of ports

#

estimated time to finish

#

nvm

#

ihnfi

#

I don't think anyone but me's ever used that abbreviation

#

and some dude named "alaxx2_damaxx"

flint idol Aug 2, 2024, 1:30 AM

#

flint idol Current roadmap: DPCs->Merge driver interface->GDB Stub->VFS->Syscalls->GCC Targ...

the vfs will make changes to the driver interface to support async io

flint idol Aug 2, 2024, 2:09 AM

#

flint idol Current roadmap: DPCs->Merge driver interface->GDB Stub->VFS->Syscalls->GCC Targ...

sometime between this (after GDB stub?), I'll implement an AHCI driver

#

after VFS I'll implement a VFAT/ext2 driver

flint idol Aug 2, 2024, 2:44 AM

#

flint idol Current roadmap: DPCs->Merge driver interface->GDB Stub->VFS->Syscalls->GCC Targ...

actually after this I might try and port OBOS to the m68k

#

#1026090868303732766 message

#

I'm just throwing that in here for tomorrow

#

when I make DPCs

flint idol Aug 2, 2024, 3:42 AM

#

https://github.com/oberrow/obos-archive/commit/4d081b345e79b20c5e7dd8e595e85cdddf9bf73c

GitHub

Fix critical allocation bug in kfindmemblock(). · oberrow/obos-arch...

#

OBOS has been having memory corruption bugs since the first rewrite lmao

#

oh my obos one was so bad

#

it took me a week or two after starting to realize I should use a build system

#

instead of a frickin' shell script

#

bruh obos #2 doesn't compile

#

it's just straight up missing a file

#

rewrite #3 has always been reliable

#

yeah that worked

#

I was able to get obos rewrite 2 to compile

#

might actually be the most unstable one

#

thank god I ditched this

#

previous rewrite has a goofy dependency system

#

so it's using the newest version of uACPI which it doesn't support

#

now I'd try and fix it

#

but I don't really care about that rewrite

#

I now go to sleep

#

(I intended to go to sleep two hours ago)

#

I swear every day I tell my self

#

go to sleep early

#

but then this kinda shit happens

real pecan Aug 2, 2024, 8:06 AM

#

lean glen doxygen when?

I've seen the quality of docs it produces, very underwhelming

ornate ginkgo Aug 2, 2024, 9:08 AM

#

flint idol actually after this I might try and port OBOS to the m68k

join us - m68k (68040+) is actually quite a fun platform.

#

I have a limine stub you can use

#

also maybe I missed something, but why do you need floating/fixed point in the kernel?

real pecan Aug 2, 2024, 9:14 AM

#

ornate ginkgo join us - m68k (68040+) is actually quite a fun platform.

i dont like that u have to assume where devices are

ornate ginkgo Aug 2, 2024, 9:14 AM

#

no me neither, I like everything being discoverable

#

but that aside, the rest of it is nice

#

I dont mean to glorify it as some perfect platform, but I enjoyed porting to it.

#

and iirc uacpi did compile, sounds like its a firmware problem that we dont have acpi there meme

real pecan Aug 2, 2024, 9:31 AM

#

lol

#

i mean uacpi works fine on 32-bit platforms so should be okay, the only problem is it assumes little endian in a few places

#

and iirc no big endian platform has acpi support(?)

ornate ginkgo Aug 2, 2024, 9:34 AM

#

so far

real pecan Aug 2, 2024, 9:35 AM

#

i mean lol

ornate ginkgo Aug 2, 2024, 9:35 AM

#

sounds like a fun weekend project tbh

real pecan Aug 2, 2024, 9:35 AM

#

if a new platform is released i doubt they'll go big endian

#

it would be DOA

ornate ginkgo Aug 2, 2024, 9:35 AM

#

oh yeah lol

#

I meant adding acpi to qemu m68k

real pecan Aug 2, 2024, 9:36 AM

#

yeah

#

wouldnt be upstreamable tho

ornate ginkgo Aug 2, 2024, 9:36 AM

#

nah of course not

real pecan Aug 2, 2024, 9:36 AM

#

because linux compiles with CONFIG_ACPI=n for this

devout niche Aug 2, 2024, 9:36 AM

#

I agree with porting to 68k

#

Great fun

#

It was considerably more fun than aarch64 has been

real pecan Aug 2, 2024, 9:37 AM

#

lol

#

aarch64 is a small indie arch so ofc

#

u wouldnt expect well tested tooling for it

#

meme

devout niche Aug 2, 2024, 9:37 AM

#

Yes not a real arch like the m68k or amd64

real pecan Aug 2, 2024, 9:37 AM

#

yeah

#

i use the m68k daily

devout niche Aug 2, 2024, 9:38 AM

#

It's the arch that matters

#

Aarch64 is nice but it's for passion projects only, not for being a relevant kernel today

ornate ginkgo Aug 2, 2024, 9:38 AM

#

lmao

real pecan Aug 2, 2024, 9:39 AM

#

devout niche Aarch64 is nice but it's for passion projects only, not for being a relevant ker...

true

#

its maintained by hobbyists that still care for it

#

whereas m68k is backed by a trillion dollar tech company and billion dollar contracts

#

so ofc

lean glen Aug 2, 2024, 11:33 AM

#

real pecan I've seen the quality of docs it produces, very underwhelming

It's as useful as the docs you put in your headersI guess

flint idol Aug 2, 2024, 1:53 PM

#

devout niche I agree with porting to 68k

wasn't keyronex initially on the 68k

#

I will be porting obos to the 68k eventually

#

idk if I should wait until I'm done more parts of the kernel or do it after the driver interface

flint idol Aug 2, 2024, 2:20 PM

#

just started working on DPCs

flint idol Aug 2, 2024, 2:21 PM

#

ornate ginkgo also maybe I missed something, but why do you need floating/fixed point in the k...

I was too lazy to make my own hashmap

#

and the library I took used floating point

#

also the timer interface uses it (rarely; only on init of a timer object) to get a time period from a frequency

#

also this:

uint64_t CoreS_TimerTickToNS(timer_tick tp)
{
    // 1/freq*1000000000*tp
    fixedptd ns = fixedpt_fromint(1); // 1.0
    const fixedptd divisor = fixedpt_fromint(CoreS_TimerFrequency); // freq
    ns = fixedpt_xdiv(ns, divisor);
    ns = fixedpt_xmul(ns, fixedpt_fromint(1000000000));
    ns = fixedpt_xmul(ns, fixedpt_fromint(tp));
    return fixedpt_toint(ns);    
}```

ashen goblet Aug 2, 2024, 2:25 PM

#

I‘m sure you can get rid of this somehow

#

I‘d certainly avoid a lot of complication with saving and resorting floating point regs

flint idol Aug 2, 2024, 2:25 PM

#

this doesn't do that

#

it's not floating point math, it's fixed point

#

it doesn't use the sse registers

#

or anything to the likes of that

#

only GPRs

#

do dpcs deserve their own object

#

hmmmm

#

yeah sure why not

#

allows me to add more convinience apis

devout niche Aug 2, 2024, 2:31 PM

#

flint idol wasn't keyronex initially on the 68k

This iteration was actually

#

It was re ported to amd64 without bother though

#

Amd64 is a pleasant arch

flint idol Aug 2, 2024, 3:24 PM

#

flint idol allows me to add more convinience apis

after failure to think of said convinience apis

#

I am not going to make DPC objects

#

DPCs will just be some struct

#

typedef struct dpc
{
    void(*handler)(struct dpc* dpc);
    bool wasRun;
    LIST_NODE(dpc_queue, struct dpc) node;
} dpc;```

#

TODO: Are dpcs in a per-cpu queue?

#

Deferred Procedure Calls (DPCs) are on a queue specific to each processor
According to ms

lean glen Aug 2, 2024, 3:31 PM

#

why do you need wasRun

#

just remove it from the list

flint idol Aug 2, 2024, 3:32 PM

#

I forgor

#

because then I don't need to check if the dpc is still in the list on free

#

I'd just check wasRun

lean glen Aug 2, 2024, 3:32 PM

#

check if its linkage is valid, thats trivial

flint idol Aug 2, 2024, 3:33 PM

#

but this is more trivial

inland radish Aug 2, 2024, 3:34 PM

#

flint idol TODO: Are dpcs in a per-cpu queue?

i would recommend doing it like that yeah

lean glen Aug 2, 2024, 3:34 PM

#

nah literally just if node.prev == NULL and node.next = NULL

flint idol Aug 2, 2024, 3:34 PM

#

and if node is at the head

#

those would be null

#

yet it would still be linked

lean glen Aug 2, 2024, 3:35 PM

#

&& list.head != node

lean glen Aug 2, 2024, 3:35 PM

#

flint idol and if `node` is at the head

*if it's the only element in the list

flint idol Aug 2, 2024, 3:35 PM

#

well yeah

#

I just saved one byte of memory*
*more because padding and stuff but whatever

#

let's go

lean glen Aug 2, 2024, 3:37 PM

#

realistically it could be one bit

#

meme

flint idol Aug 2, 2024, 3:38 PM

#

#define LIST_IS_NODE_UNLINKED(name, list, node)\
(LIST_GET_HEAD(list) != node && LIST_GET_NEXT(node) == nullptr && LIST_GET_PREV(node) == nullptr)```

#

both the dpc helper functions and the dpc dispatching thing is done

#

if (to < IRQL_DISPATCH)
{
    // Run pending DPCs on the current CPU.
    for (dpc* cur = LIST_GET_HEAD(dpc_queue, &CoreS_GetCPULocalPtr()->dpcs); cur; )
    {
        dpc* next = LIST_GET_NEXT(dpc_queue, &CoreS_GetCPULocalPtr()->dpcs, cur);
        cur->handler(cur, cur->userdata);
        LIST_REMOVE(dpc_queue, &CoreS_GetCPULocalPtr()->dpcs, cur);
        cur = next;
    }
}```

#

I put this code in lower irql

#

to dispatch the DPCs

#

which I think is right

#

this is the final dpc struct:

typedef struct dpc
{
    LIST_NODE(dpc_queue, struct dpc) node;
    void(*handler)(struct dpc* dpc, void* userdata);
    void* userdata;
    struct cpu_local* cpu;
} dpc;```

#

bruh my graphics driver keeps on dying

#

nvm the plug to my monitor was just disconnected

#

I've changed my com irq handler to use DPCs

flint idol Aug 2, 2024, 4:17 PM

#

flint idol ```c if (to < IRQL_DISPATCH) { // Run pending DPCs on the current CPU. f...

I'm pretty sure this is supposed to run before the irql change

#

oops

#

IRQL on call of the dispatcher is less than the IRQL of the vector reported by the architecture ("irql_ <= Core_GetIrql()").

#

KEKW

#

must be spooky actions from afar

#

t'was a race condition

#

now I can defer IRQs*

#

*I need to fix a bug where it defers right before the IRQ dispatcher exits

#

actually maybe that wouldn't be a problem

#

should I defer the irq dispatcher

#

~~even better, should I defer the DPC dispatcher~~

#

I will now make the timer irq use DPCs

#

instead of signalling a thread

#

I also made the kernel api for uACPI properly defer work

#

instead of using a thread

#

DPCs are amazing

#

I just need to make the timer irq use DPCs

#

then I can merge the driver interface

#

how cool right

#

I should probably make the timer dispatcher better soon

#

it just searches some linked list for timers that expired

#

which isn't neccessarily fast

#

oof

#

I get a use after free in the vmm

#

I love when I change something in the kernel

#

then some completely unrelated component breaks

flint idol Aug 2, 2024, 5:49 PM

#

ok I fixed it

#

seems to be a bad idea to make a new DPC on every single timer irq

#

maybe I just did something wrong though

#

TODO: fix

#

the driver interface is done

#

I'm merging it now

#

time to make a gdb stub

#

this'll be fun*

#

*not really

flint idol Aug 2, 2024, 8:40 PM

#

why the fuck is obos like this

#

I add code (completely unrelated to the uart driver or really anything)

#

and it fucking crashes

#

oh of course it decides to fail in PnP

#

I hate C

#

#

conviniently, adding -Werror=incompatible... compiles fine

flint idol Aug 2, 2024, 9:55 PM

#

hmmm something werid is happening

#

the com driver doesn't seem to like the character '$'

flint idol Aug 2, 2024, 10:14 PM

#

for some reason the com driver likes discarding the first character sent

#

but only the very first

#

it's not discarded in the irq handler though

#

it's discarded on read from the input buffer

#

otherwise receiving/sending packets work

flint idol Aug 2, 2024, 10:43 PM

#

I fixed bugs with receiving packets

#

which means I can get started on the rest of the stub

#

it can also receive a packet from gdb

#

it would be packets but I only have 1 call to the receive function

#

I think I'll make there be some master dispatcher that receives commands from gdb and calls a handler for them

#

or otherwise just sends nothing to gdb

#

because iirc when you don't support a gdb stub request you just reply with nothing and gdb will recognize that it's unsupported

flint idol Aug 2, 2024, 10:48 PM

#

flint idol I think I'll make there be some master dispatcher that receives commands from gd...

the handlers will be in a hashmap

#

because hashmaps are cool

#

O(1) best lookup time is amazing for stuff like this

#

and assuming the hashing algorithm isn't absolute doo doo, it's pretty often you get O(1) lookup

#

anyway enough yapping, more making dispatcher

flint idol Aug 3, 2024, 12:07 AM

#

this is amazing

#

using a gdb stub you can query symbols and stuff qSymbol:sym_name

#

using that command

#

I have finished parsing command names

#

I can now start implementation of commands

flint idol Aug 3, 2024, 12:18 AM

#

flint idol it's discarded on read from the input buffer

I'm stupid

#

it was read in the send function

#

because it wanted an acknoledgement

flint idol Aug 3, 2024, 1:56 AM

#

my strchr is broken

#

strchr:
    push rbp
    mov rbp, rsp

    push rdi
    call strlen
    pop rdi
    mov rcx, rax
    mov al, sil
    repne scasb
    mov rax, rcx
    dec rax

    leave
    ret```

#

maybe I'm supposed to use repe

#

nah

flint idol Aug 3, 2024, 2:21 AM

#

oh wait

#

I can now respond to gdb packets

#

while it is a response with nothing

#

since I support nothing

#

it's still something

#

TODO:
Implement ? packet
Implement qC packet
Implement a bunch of other packets

ornate ginkgo Aug 3, 2024, 8:03 AM

#

flint idol ```c if (to < IRQL_DISPATCH) { // Run pending DPCs on the current CPU. f...

shouldnt matter, but you probably want to remove the in-flight dpc from the list before calling its handler.

main girder Aug 3, 2024, 8:31 AM

#

does matter

#

you MUST do that

ornate ginkgo Aug 3, 2024, 8:31 AM

#

I havent run into a situation where it mattered, but im curious as to your reasoning?

main girder Aug 3, 2024, 8:32 AM

#

otherwise you violate a guarantee that when you enqueue a DPC it will start executing from the beginning at some point in the future

#

which creates race conditions where you might receive an interrupt and the handler enqueues a DPC which is currently executing and it was already enqueued so that's a no op

#

but the DPC, being halfway done, missed whatever event you wanted to signal it of

#

if you dequeue the DPC before running the routine, and this happens, then you will re-enqueue it and it will absolutely for sure see all the state you want it to see on the next time it runs

ornate ginkgo Aug 3, 2024, 8:34 AM

#

makes sense, I feel like that could be dealt with by polling within the dpc

main girder Aug 3, 2024, 8:34 AM

#

wdym

#

exit the polling loop -> leave the dpc routine -> dequeue it
between first and last step this race condition can occur

ornate ginkgo Aug 3, 2024, 8:35 AM

#

yeah ok, fair

main girder Aug 3, 2024, 8:35 AM

#

you could disable interrupts maybe before exiting the polling loop but then that defeats the purpose of minimizing time with interrupts disabled

ornate ginkgo Aug 3, 2024, 8:36 AM

#

yeah lol

main girder Aug 3, 2024, 8:36 AM

#

so it's best to just dequeue it before you run it

#

so it can be enqueued again while running

ornate ginkgo Aug 3, 2024, 8:36 AM

#

its also cleaner, keep your state tidy before branching out to other code

main girder Aug 3, 2024, 8:36 AM

#

main girder so it can be enqueued again while running

(& it won't run again til it's done the first time)

#

another detail about DPCs is that you should only enqueue one to the same processor as last time you enqueued it

#

for that reason

#

usually you get this automatically by only handling an interrupt on one core and then when it enqueues its dpc it enqueues it to the current core which is the same one every time

#

if it's a situation where you get the same interrupt in parallel (like an interval timer that you've configured to broadcast its irq to all cores) and you want it to enqueue a dpc, you need to have one of those dpcs per core

ornate ginkgo Aug 3, 2024, 8:42 AM

#

good info, thanks

main girder Aug 3, 2024, 8:43 AM

#

In theory you could add another lock to the dpc itself or something to guard against enqueuing the same one from multiple processors concurrently

#

But that's an extra lock (which is likely to be contended, from interrupt context) which is lame

ornate ginkgo Aug 3, 2024, 8:43 AM

#

that just sounds like moving the problem

flint idol Aug 3, 2024, 10:05 AM

#

hmm ok

#

I'll do that

#

now that I think of it I get what you mean

#

I assume I should error out in the enqueue dpc function if it's already enqueued

main girder Aug 3, 2024, 10:22 AM

#

it should just be a no-op

#

you can return a status saying it wasn't enqueued but it shouldn't crash or anything

flint idol Aug 3, 2024, 10:22 AM

#

it doesn't crash

#

it just returns a status

#

OBOS_STATUS_DPC_ALREADY_ENQUEUED

main girder Aug 3, 2024, 10:28 AM

#

Why not just TRUE or FALSE

flint idol Aug 3, 2024, 10:28 AM

#

because I like status codes

#

no harm in returning them

#

instead of a bool

real pecan Aug 3, 2024, 10:31 AM

#

main girder Why not just TRUE or FALSE

because it can fail for other reasons

#

and its nice to know why

flint idol Aug 3, 2024, 10:32 AM

#

(unlike my previous kernel, where you'd just get a cryptic true or false if you're lucky)

flint idol Aug 3, 2024, 12:19 PM

#

I pushed the code for sending gdb packets

#

I've implemented the packets:
qC
qfThreadInfo/qsThreadInfo

flint idol Aug 3, 2024, 12:41 PM

#

bruh\

#

the hashmap stopped working

#

oops

#

still doesn't work

#

the hashes are the same...

#

I'm stupid

#

my strcmp returns bool

#

and I was using it in cmp_packet

#

which expected the normal strcmp return value

flint idol Aug 3, 2024, 1:16 PM

#

I'm currently going to implement debug/breakpoint exceptions

#

so that I can send register context to gdb

#

I think I want to defer the exceptions though and block the current thread in the handlers

#

instead of handling a bunch of gdb packets at irql_masked

#

or some irql that's high-priority

#

I think that now I have DPCs I'm going abuse the hell out of them

flint idol Aug 3, 2024, 2:23 PM

#

I just realized my uart driver's read_sync function is actually async

#

since if there's not enough characters in the input buffer, instead of blocking, it returns

#

// TODO: Make sync instead of async.

#

that should be good enough

#

it isn't good enough

#

actually no

flint idol Aug 3, 2024, 3:02 PM

#

the thing is being weird

#

there is an inbound irq from the ioapic to cpu 0

#

it's in the IRR of the ioapic

#

but not for the LAPIC

#

despite eflags.if being one and cr8 < the priority of the irq

flint idol Aug 3, 2024, 3:55 PM

#

this math ain't mathin'

#

the irql is fine

#

eflags.if is true

#

ISR in the LAPIC has nothing, so it can't be that an EOI wasn't send after an IRQ

#

IRR in the LAPIC doesn't have the IRQ

#

IRR in the IOAPIC does though

#

maybe it's being received

#

but something weird is happening so it's in the ioapic irr register

#

hmm

#

it's received

#

except only on the first four

#

then it just isn't received no more

#

maybe it's waiting for the buffer to be read from

#

before allowing the irq to pass

#

(for reference this is for the UART driver)

#

// TODO: Is this a good idea?

#

I don't think it was

#

ok I fixed it

main girder Aug 3, 2024, 4:22 PM

#

real pecan because it can fail for other reasons

DPC enqueue?

#

It shouldn't be able to fail for other reasons than already being enqueued lol

flint idol Aug 3, 2024, 4:22 PM

#

invalid argument

main girder Aug 3, 2024, 4:23 PM

#

main girder It shouldn't be able to fail for other reasons than already being enqueued lol

That's kind of an important aspect of them

flint idol Aug 3, 2024, 4:23 PM

#

I think infy was talking about generally

#

having a status code is better than having a plain bool because it's more descriptive

real pecan Aug 3, 2024, 4:23 PM

#

oom, null pointer, other shit potentially

flint idol Aug 3, 2024, 4:23 PM

#

null ptr would be an invalid arg

#

oom doesn't exist

main girder Aug 3, 2024, 4:24 PM

#

real pecan oom, null pointer, other shit potentially

No

flint idol Aug 3, 2024, 4:24 PM

#

since the dpc enqueue function doesn't allocate

main girder Aug 3, 2024, 4:24 PM

#

You cannot oom on dpc enqueue that would imply a horrid broken dependency on the allocator

real pecan Aug 3, 2024, 4:24 PM

#

using true/false to return a status code because "dude it can only fail because X trust me" is not a valid argument

#

its annoying

main girder Aug 3, 2024, 4:24 PM

#

And if you pass a null pointer to an internal function that's a bug

main girder Aug 3, 2024, 4:24 PM

#

real pecan using true/false to return a status code because "dude it can only fail because ...

It is a valid argument

#

With precedent

#

It's not trust me bro it's rigorous and well understood why it must be that way

real pecan Aug 3, 2024, 4:25 PM

#

lol

main girder Aug 3, 2024, 4:25 PM

#

main girder With precedent

That's how it is on NT and to this day the only failure mode for DPC enqueue is that

#

That it was already enqueued

lean glen Aug 3, 2024, 4:25 PM

#

just use monadic types for error handling like a sane person

real pecan Aug 3, 2024, 4:25 PM

#

well we dont know because we dont have nt source

flint idol Aug 3, 2024, 4:26 PM

#

but don't we have docs for nt

main girder Aug 3, 2024, 4:26 PM

#

We do know because that function is exported

#

If DPC enqueue could fail for other reasons the system would be broken because there's no good way to recover from that

real pecan Aug 3, 2024, 4:26 PM

#

main girder We do know because that function is exported

dpc enqueue function returns a bool?

main girder Aug 3, 2024, 4:26 PM

#

It's like saying an interrupt handler could randomly fail to be called

real pecan Aug 3, 2024, 4:27 PM

#

https://learn.microsoft.com/en-us/windows-hardware/drivers/ddi/wdm/nf-wdm-keinsertqueuedpc

KeInsertQueueDpc function (wdm.h) - Windows drivers

The KeInsertQueueDpc routine queues a DPC for execution.

#

this one?

flint idol Aug 3, 2024, 4:27 PM

#

~~well technically if the handler is nullptr it'd pf on call~~

main girder Aug 3, 2024, 4:27 PM

#

https://learn.microsoft.com/en-us/windows-hardware/drivers/ddi/wdm/nf-wdm-keinsertqueuedpc

KeInsertQueueDpc function (wdm.h) - Windows drivers

The KeInsertQueueDpc routine queues a DPC for execution.

#

Yeah that

real pecan Aug 3, 2024, 4:27 PM

#

flint idol ~~well technically if the handler is nullptr it'd pf on call~~

it would df

main girder Aug 3, 2024, 4:27 PM

#

Why

flint idol Aug 3, 2024, 4:27 PM

#

what's df

main girder Aug 3, 2024, 4:27 PM

#

Double fault

real pecan Aug 3, 2024, 4:27 PM

#

double fault

#

inability to call an idt handler is a double fault

#

e.g. if it dies while trying to push to a stack

main girder Aug 3, 2024, 4:28 PM

#

Infy the only thing dpc enqueue does is insert it into an intrusively linked list and set a software interrupt pending at DISPATCH_LEVEL which is basically just setting a bit somewhere depending on how you implement the software levels

#

You are wrong on this one

real pecan Aug 3, 2024, 4:29 PM

#

ok ok if thats what nt does then i believe it

flint idol Aug 3, 2024, 4:29 PM

#

wait wat's that last part

real pecan Aug 3, 2024, 4:29 PM

#

but im still against using bool

main girder Aug 3, 2024, 4:29 PM

#

real pecan ok ok if thats what nt does then i believe it

You should believe it even if NT didn't do it

flint idol Aug 3, 2024, 4:29 PM

#

bruh my checksum function might not be working

main girder Aug 3, 2024, 4:30 PM

#

DPCs would be completely useless if they could fail

real pecan Aug 3, 2024, 4:30 PM

#

memory allocation would be useless if it could fail as well meme

main girder Aug 3, 2024, 4:30 PM

#

They are semantically interrupt handlers and you enqueue them to finish up work in your ISR that can't be performed at that higher level

#

Because spinlocks are acquired at DISPATCH_LEVEL in the rest of the system to avoid blocking out hardware interrupts

real pecan Aug 3, 2024, 4:31 PM

#

makes sense

main girder Aug 3, 2024, 4:31 PM

#

They are DEFINED to never fail

#

If they could then your driver could randomly find itself unable to mark an IO packet completed and things like that

real pecan Aug 3, 2024, 4:32 PM

#

whats the diff between normal and threaded dpc

flint idol Aug 3, 2024, 4:32 PM

#

one has a fancy word added to it

real pecan Aug 3, 2024, 4:33 PM

#

main girder If they could then your driver could randomly find itself unable to mark an IO p...

yeah i get it

main girder Aug 3, 2024, 4:33 PM

#

Threaded DPCs switch to a per cpu dpc dispatch thread which frees up the previously running thread to get picked up by other cores

#

It's good for if your DPC is really lengthy for some reason

#

It shouldn't be but some driver authors are dumb

real pecan Aug 3, 2024, 4:33 PM

#

frees up the previously running thread to get picked up by other cores
i didnt understand any of that

flint idol Aug 3, 2024, 4:34 PM

#

it means that the current thread on that cpu ceases to run on that core

main girder Aug 3, 2024, 4:34 PM

#

DPCs are basically interrupt handlers which run implicitly in the context of whatever thread was currently running

flint idol Aug 3, 2024, 4:34 PM

#

flint idol it means that the current thread on that cpu ceases to run on that core

gets put on another one

main girder Aug 3, 2024, 4:34 PM

#

As long as that thread is still marked as running on that CPU it can't be run on any others

flint idol Aug 3, 2024, 4:34 PM

#

flint idol gets put on another one

then that core switches to some dpc-dispatching thingamajig thread

main girder Aug 3, 2024, 4:34 PM

#

So you switch away from its context before executing the DPCs which causes it to be placed on the ready queues

#

Where it can be picked up and run by another core

#

Rather than waiting for the DPCs to complete execution

real pecan Aug 3, 2024, 4:35 PM

#

so like a separate thread?

main girder Aug 3, 2024, 4:35 PM

#

Wdym

real pecan Aug 3, 2024, 4:35 PM

#

like executing a function in a thread pool

main girder Aug 3, 2024, 4:35 PM

#

Not exactly

#

Because the DPCs still aren't preemptible and they still can't take blocking mutexes or anything

real pecan Aug 3, 2024, 4:36 PM

#

sooo

#

the difference is its a different stack or?

#

from normal dpc

main girder Aug 3, 2024, 4:36 PM

#

The difference is the thread you were running can keep running on another core while the dpc is running

#

And that's the mechanism for that yeah

real pecan Aug 3, 2024, 4:37 PM

#

i see

flint idol Aug 3, 2024, 4:37 PM

#

I'll implement threaded DPCs soon enough

real pecan Aug 3, 2024, 4:37 PM

#

linux doesnt have dpcs at all

#

just work queues

#

and soft irqs

flint idol Aug 3, 2024, 4:38 PM

#

common linux L

real pecan Aug 3, 2024, 4:38 PM

#

like bottom halves

flint idol Aug 3, 2024, 4:38 PM

#

also what's an APC

#

is it like a dpc

main girder Aug 3, 2024, 4:38 PM

#

If a DPC is lengthy enough to warrant switching away from the current thread it should have been a worker thread item

flint idol Aug 3, 2024, 4:38 PM

#

flint idol is it like a dpc

but on a thread

main girder Aug 3, 2024, 4:38 PM

#

But some driver authors are stupid like I said

main girder Aug 3, 2024, 4:39 PM

#

flint idol also what's an APC

You can enqueue them to a thread to get that thread to execute an arbitrary function in its context next time it's scheduled in

flint idol Aug 3, 2024, 4:40 PM

#

would it be any arbitrary thread

main girder Aug 3, 2024, 4:40 PM

#

Wdym

flint idol Aug 3, 2024, 4:40 PM

#

like would the kernel choose any arbitrary thread to enqueue the apc onto

real pecan Aug 3, 2024, 4:40 PM

#

u choose the thread

#

and the fn gets executed next time the thread gets to run

main girder Aug 3, 2024, 4:41 PM

#

Yeah the point is it runs in the context of some specific thread

#

NT does this for the latter part of IO request completion

real pecan Aug 3, 2024, 4:42 PM

#

like deferred dpc with thread affinity

main girder Aug 3, 2024, 4:42 PM

#

It enqueues an APC to the requesting thread

main girder Aug 3, 2024, 4:42 PM

#

real pecan like deferred dpc with thread affinity

Big difference is that the APC can page fault and take locks and stuff

#

NT uses them to access the userspace of the requesting thread to copy out the "status block" for the request

#

To its userspace

real pecan Aug 3, 2024, 4:42 PM

#

i see

#

whats the status block

main girder Aug 3, 2024, 4:43 PM

#

Just contains the status for the operation and other information like number of bytes written or read

#

This is for async io requests

real pecan Aug 3, 2024, 4:43 PM

#

they literally run in userspace?

main girder Aug 3, 2024, 4:43 PM

#

Huh?

#

No

#

Kernel APCs don't

#

But while they're running the thread's process's userspace is visible in the lower half

real pecan Aug 3, 2024, 4:43 PM

#

oh u mean it runs with the cr3 of that process

#

so that it can access it

main girder Aug 3, 2024, 4:44 PM

#

Yeah

real pecan Aug 3, 2024, 4:44 PM

#

i see

main girder Aug 3, 2024, 4:44 PM

#

When you start the IO request you give it a pointer to where you want the status block to be copied out

#

So you can have like 20 of them inflight and this info gets copied out asynchronously in the context of your thread as they complete and then you can check on it later

#

There are also usermode APCs which are mechanically a lot like unix signals

#

You can optionally supply a pointer to a callback function to be executed in the context of your thread when an IO request completes

real pecan Aug 3, 2024, 4:45 PM

#

whats the syscall to request an async read for example?

main girder Aug 3, 2024, 4:46 PM

#

main girder You can optionally supply a pointer to a callback function to be executed in the...

And this is implemented via enqueuing a usermode APC to your thread (from the kernel mode APC that was doing IO completion)

main girder Aug 3, 2024, 4:46 PM

#

real pecan whats the syscall to request an async read for example?

NtReadFile with relevant flags

real pecan Aug 3, 2024, 4:46 PM

#

and how do u get the pointer to the status block

main girder Aug 3, 2024, 4:46 PM

#

You can also have it be synchronous and a lot of this is bypassed for efficiency and you don't have to do an extra syscall to wait

main girder Aug 3, 2024, 4:46 PM

#

real pecan and how do u get the pointer to the status block

You give it the pointer to the status block

#

You allocate that memory yourself before you start the request

real pecan Aug 3, 2024, 4:47 PM

#

#

ohhh jesus

#

wait what

main girder Aug 3, 2024, 4:48 PM

#

It's talking about using NtReadFile from kernel mode

#

Usermode usage of the native API is undocumented

#

But calling that from usermode that's where it passes the callback pointer and a context word

#

There's a win32 wrapper api for this

#

ReadFileAsync or something idk

real pecan Aug 3, 2024, 4:49 PM

#

oh

main girder Aug 3, 2024, 4:49 PM

#

main girder It's talking about using NtReadFile from kernel mode

NT is designed so all of the system services can be called from kernel mode (and then they're just direct calls of the entrypoint)

real pecan Aug 3, 2024, 4:50 PM

#

and from kernel mode u cant provide an apc?

main girder Aug 3, 2024, 4:50 PM

#

This was a specific thing they did to be better than unix where attempting to like open and read a file from kernel context was almost impossible

real pecan Aug 3, 2024, 4:50 PM

#

i see

main girder Aug 3, 2024, 4:51 PM

#

real pecan and from kernel mode u cant provide an apc?

If you need an asynchronous completion callback you'd probably use some other API

real pecan Aug 3, 2024, 4:51 PM

#

like non-usermode api?

main girder Aug 3, 2024, 4:51 PM

#

Probably an Io one or something

#

Yeah

real pecan Aug 3, 2024, 4:51 PM

#

i see

main girder Aug 3, 2024, 4:52 PM

#

main girder This was a specific thing they did to be better than unix where attempting to li...

Unix did this insane shit where it had this per process "uarea" where it'd stash syscall arguments and stuff rather than passing them as arguments up and down the call stack in the kernel

#

So if you wanted to call a syscall from kernel mode you needed to save and restore extremely implementation and even architecture dependent fields of the uarea

flint idol Aug 3, 2024, 4:53 PM

#

that's insane

real pecan Aug 3, 2024, 4:53 PM

#

yeah its definitely bad design

flint idol Aug 3, 2024, 4:53 PM

#

what were they on

#

they were definitely on some funky shit ngl

main girder Aug 3, 2024, 4:54 PM

#

It was because on the pdp11 their kernel stack was like 512 bytes and they only had one

lean glen Aug 3, 2024, 4:54 PM

#

remember this is the 70s

#

hardware was limited

#

yeah this

main girder Aug 3, 2024, 4:54 PM

#

So they couldn't save anything on the kernel stack across blocking calls

#

So they'd stash it in the uarea

main girder Aug 3, 2024, 4:54 PM

#

lean glen remember this is the 70s

No even by 1989 most unix variants were still doing this

#

That's when NT was designed

real pecan Aug 3, 2024, 4:55 PM

#

how do u know if unix was proprietary

flint idol Aug 3, 2024, 4:55 PM

#

🕵️‍♂️

lean glen Aug 3, 2024, 4:55 PM

#

there were open source unixes kinda

main girder Aug 3, 2024, 4:55 PM

#

Because of infinity contemporary sources

lean glen Aug 3, 2024, 4:55 PM

#

like lions book had the source code didnt it

main girder Aug 3, 2024, 4:56 PM

#

I can link you a video of a DEC guy presenting the design of NT to other DEC guys where at one point he says it's better than unix for this exact reason lol

#

He recites an anecdote of trying to open and read a file from inside a driver on ultrix

#

And ultrix was highly representative of the state of unix at the time

real pecan Aug 3, 2024, 4:56 PM

#

i think u linked it before

main girder Aug 3, 2024, 4:56 PM

#

lean glen there were open source unixes kinda

The bell labs unix versions have been mostly open sourced

#

They are from the 70s though

#

There is also svr4 source on the internet which is technically not open but nobody is using it anymore, it's not even clear who owns it iirc, and it's decades out of date

#

So if you wanted to look and not tell anyone you'd be safe probably

flint idol Aug 3, 2024, 4:58 PM

#

what'd happen if you tell someone

#

would the fbi be at your door

#

within two minutes

main girder Aug 3, 2024, 4:59 PM

#

flint idol what'd happen if you tell someone

Most likely nothing ever

#

But certain people are paranoid as fuck and would get upset with you for admitting doing that

flint idol Aug 3, 2024, 4:59 PM

#

ok

#

some funky stuff are happening with DPCs rn

#

what: use after free while reading a dpc

#

why: the node wasn't removed properly

#

how: idk

#

where: in the dpc dispatcher

#

(ok I don't need to do who what where when why)

real pecan Aug 3, 2024, 5:08 PM

#

how: skill issue

flint idol Aug 3, 2024, 5:11 PM

#

I think there might be a stack overflow of some sorts

main girder Aug 3, 2024, 5:12 PM

#

Send source

#

I'll look in a bit

flint idol Aug 3, 2024, 5:12 PM

#

ok just lemme update the repo

#

well it isn't a stack overflow

#

but there is infinite recursion

#

ok pushed

#

https://github.com/OBOS-dev/obos/tree/gdbstub

GitHub

GitHub - OBOS-dev/obos at gdbstub

I was bored... Contribute to OBOS-dev/obos development by creating an account on GitHub.

#

kernel is under src/oboskrnl

main girder Aug 3, 2024, 5:19 PM

#

Is this a uniprocessor system

flint idol Aug 3, 2024, 5:19 PM

#

no

#

SMP

#

the infinite recursion goes away when I don't use DPCs for the com irq

#

when I use DPCs, but make the irq dispatcher call the variant of lower irql that doesn't touch DPCs, nothing happens

#

probably because the irql is never lowered below IRQL_DISPATCH with LowerIrql

#

(note: it is being lowered below it, except only on ctx switch in the scheduler)

#

hold up\

#

DPCs are being executed at passive level

main girder Aug 3, 2024, 5:30 PM

#

Man you messed up executing functions on a list

flint idol Aug 3, 2024, 5:30 PM

#

originally I did

#

I must've accidentally removed the code

#

yeah I accidentally removed that code

real pecan Aug 3, 2024, 5:33 PM

#

main girder Man you messed up executing functions on a list

lol

flint idol Aug 3, 2024, 5:35 PM

#

ok everything works now

#

(except for the response to the qSupported packet)

flint idol Aug 3, 2024, 5:51 PM

#

qSupported is now supported

#

so is qC, q*ThreadInfo, and qAttached

#

TODO: QStartNoAckMode, qRcmd

main girder Aug 3, 2024, 5:53 PM

#

Your allocate and free dpc functions are silly btw

flint idol Aug 3, 2024, 5:54 PM

#

oof

main girder Aug 3, 2024, 5:54 PM

#

Especially free which is broken

#

Allocate is just unnecessary

flint idol Aug 3, 2024, 5:55 PM

#

main girder Especially free which is broken

broken or named wrong

flint idol Aug 3, 2024, 5:55 PM

#

main girder Allocate is just unnecessary

that's just a helper

main girder Aug 3, 2024, 5:55 PM

#

Broken

main girder Aug 3, 2024, 5:56 PM

#

flint idol that's just a helper

To help you be bad?

main girder Aug 3, 2024, 5:56 PM

#

main girder Broken

You account for if it's in the queue but what if it's in the queue of another processor or already running on another processor

#

I don't see the synchronization here

flint idol Aug 3, 2024, 5:57 PM

#

I removed the per-cpu dpc queue lock after I thought that was causing problems (it wasn't)

main girder Aug 3, 2024, 5:57 PM

#

main girder To help you be bad?

In general it's bad to do tons of little allocations like this both because the heap metadata takes up more space and cuz it's not very efficient

#

You should just statically allocate dpc structures or place them inline into other structures

main girder Aug 3, 2024, 5:58 PM

#

flint idol I removed the per-cpu dpc queue lock after I thought that was causing problems (...

A lock still doesn't help with if the DPC is currently running

#

Unless you put the lock around the entire act of dequeuing and executing the DPC in which case L that's terrible

#

You also need to disable interrupts entirely while manipulating the DPC queue since they can be enqueued from interrupts of any priority level

flint idol Aug 3, 2024, 6:00 PM

#

ok

#

anyway I'll be back in an hour

flint idol Aug 3, 2024, 7:02 PM

#

ok I'm back

flint idol Aug 3, 2024, 9:09 PM

#

maybe I just shouldn't be deferring the com irq

#

because otherwise there's a chance another com irq comes

#

and gets ignored because there is already a com dpc running

#

leading to data loss at best

main girder Aug 3, 2024, 9:10 PM

#

Not how it works tho

flint idol Aug 3, 2024, 9:10 PM

#

I am losing irqs

#

though

#

which cause the kernel to hang while waiting for data from gdb in the gdb stub

main girder Aug 3, 2024, 9:11 PM

#

Why are you losing irqs

flint idol Aug 3, 2024, 9:13 PM

#

basically if the com irq is entered while the com dpc is running, it doesn't schedule a new dpc, so the irq is just lost

#

then for some reason the uart decides to give up on sending irqs after a bit

main girder Aug 3, 2024, 9:14 PM

#

You did it wrong

flint idol Aug 3, 2024, 9:14 PM

#

I could just be doing something wrong

#

yeah

main girder Aug 3, 2024, 9:15 PM

#

I explained this already I think

flint idol Aug 3, 2024, 9:17 PM

#

I think here #1026090868303732766 message

#

how the hell am I failing calling a list of functions

flint idol Aug 3, 2024, 10:46 PM

#

I have implemented the g packet

#

!!!

#

I can successfully connect to gdb

#

without it crashing

#

it can query the threads

#

and registers

#

but can't do anything else

#

I've implemented the k packet

#

which kills the process (in my case, it politely asks uacpi to shutdown)

#

now to implement useful packets

flint idol Aug 3, 2024, 11:08 PM

#

I need to start reading the docs more carefully

#

I just implemented the H packet

#

but it parses the thread immediately and sends the response

flint idol Aug 3, 2024, 11:46 PM

#

I fixed the H packet and implement the T packet

#

now for the m packet

flint idol Aug 4, 2024, 1:13 AM

#

I'm done implementing the m packet

#

for reading memory

#

(it was pain)

#

it's so annoying since I have to first verify the memory exists

#

then I need to adjust the size

#

based on the amount of memory that exists

#

then I need to format some very hex string

#

with the memory

#

I need to implement:

the M packet (writes to memory)
the s packet (steps one instruction)
the G packet (writes registers)
the vCont packet (continues/steps, but in a multithreaded way)
the qRcmd packet (used by the monitor gdb command)
the c packet (continues the program)

#

I'll use qRcmd to view kernel structs

flint idol Aug 4, 2024, 1:41 AM

#

flint idol I need to implement: - the `M` packet (writes to memory) - the `s` packet (steps...

before that, I need to implement proper stopping of the kernel

#

on int3/debug exception

flint idol Aug 4, 2024, 2:09 AM

#

flint idol how the hell am I failing calling a list of functions

wait I think I was supposed to make some sort of software interrupt

#

so that the dpc runs

#

or something

#

maybe I'm just pulling that outta nowhere though

main girder Aug 4, 2024, 2:11 AM

#

how are you running them currently

flint idol Aug 4, 2024, 2:12 AM

#

whenever irql is lowered to < IRQL_DISPATCH, the kernel dispatches all the DPCs on the current cpu's queue

main girder Aug 4, 2024, 2:12 AM

#

thats a software interrupt

#

you are simulating it

#

in software

flint idol Aug 4, 2024, 2:14 AM

#

ok then

main girder Aug 4, 2024, 2:15 AM

#

yeppers

#OBOS (not vibecoded)