OBOS (not vibecoded) | Operating System Development | Page 3

flint idol Jan 17, 2024, 9:31 PM

#

m_drawingBuffer->addr is a uint32_t*

flint idol Jan 18, 2024, 12:32 AM

#

2000th message

flint idol Jan 18, 2024, 10:41 PM

#

I fixed scrolling

#

I don't remember how, but I did

astral mason Jan 18, 2024, 10:45 PM

#

Yay! Always great when stuff works.

flint idol Jan 18, 2024, 10:45 PM

#

I think it was a missing call to SwapBuffers

#

in the scrolling code

#

idk I need to check git changes

flint idol Jan 18, 2024, 11:57 PM

#

Fuck there's memory corruption in the AHCI driver

#

and the kernel is hanging

#

And the cd thingy isn't opening

#

Ok the cd thingy opens

flint idol Jan 19, 2024, 1:12 AM

#

sigh heap corruption

#

AGAIN

#

nvm just me forgetting to memset some memory to zero

flint idol Jan 19, 2024, 1:44 AM

#

There seems to be a bug with the FAT driver with multiple FAT partitions

#

Idk where it is

#

But all I know is that it has to do with the same files being listed for two mount points that are backed by seperate partitions

#

It could be mount

#

it could be the fat driver's file iterator functions

#

It could be anything

flint idol Jan 20, 2024, 12:41 AM

#

I'm sick of no debugger on real hardware

#

I found a PC for ~140 CAD with a serial port

#

It has an ivy bridge intel processor

#

1 TB SATA Hard Disk

#

8 GB of RAM

#

It'd be perfect for debugging the OS

#

It even has PS/2

#

Natively

#

It's surprising, however, that so much people seem to be buying this PC in ~2022 even though it has a processor from 2012

lean glen Jan 20, 2024, 1:00 AM

#

it's a 10 years old cpu

flint idol Jan 20, 2024, 1:00 AM

#

For what the computer I might buy

#

My development computer's CPU is from 2013

flint idol Jan 20, 2024, 1:37 AM

#

I am a fucking idiot

#

struct pair
{
    T1 first;
    T2 second;
    bool operator==(const pair& other) const
    {
        return 
            other.first == other.first &&
            other.second == other.second;
    }
};```

#

The bug was because of this

#

and I was using a pair as a key in the hashmap for FAT32 partitions

#

In FileIteratorCreate, it gets the partition for the file iterator from the hashmap

#

and like everywhere else

#

I can read a file on real hardware

#

That took long enough

#

From the start of this kernel (October 2023) to now

#

Time to stress test it

#

Ok I stress tested it the same way as I did on qemu

#

It works

#

What should I do now, implement Write or work on syscalls

#

Write would be nice

#

But I've been saying I'm going to implement syscalls and stuff for a long time...

#

There is also a bug with my buffer swapping code or scrolling code

#

It has to do with the framebuffer's pitch

flint idol Jan 20, 2024, 2:04 AM

#

flint idol But I've been saying I'm going to implement syscalls and stuff for a long time.....

I think I'll merge disk-related-driver-dev with master and start syscalls

flint idol Jan 20, 2024, 4:50 PM

#

Done implementing handles and thread handles

#

For syscalls

#

Time to make file handles and drive handles for syscalls

#

Then console syscalls

#

Then user space programs?

#

Possibly before userspace programs I'll add a process handle syscall

flint idol Jan 20, 2024, 8:39 PM

#

Of course the scheduler HAS to stop working

#

When I'm trying to debug anything

#

Because the kernel has to be annoying, the scheduler is only being called on cpu 3

#

interrupts are enabled on other cpus, and the timer interrupt isn't masked

#

on those cpus

#

When in doubt, rebuild

ashen goblet Jan 20, 2024, 8:51 PM

#

flint idol When in doubt, rebuild

How long does rebuilding take for you

flint idol Jan 20, 2024, 8:51 PM

#

A few seconds

#

It's "only" 80 files to build

#

I triple fault however, while going into userspace

ashen goblet Jan 20, 2024, 8:53 PM

#

Ooof

flint idol Jan 20, 2024, 8:53 PM

#

I think

#

I actually think it somehow page faults while writing to the framebuffer

ashen goblet Jan 20, 2024, 8:54 PM

#

I‘m not even there yet i‘m trying to figure out how tf to do proper scheduling and task switching and threads

flint idol Jan 20, 2024, 8:55 PM

#

Ok good luck

ashen goblet Jan 20, 2024, 8:57 PM

#

I think I‘ll just try to get some minor things working since I‘m having trouble to understand the greater picture

#

Whatever I shouldn’t be littering your thread

flint idol Jan 20, 2024, 9:00 PM

#

ok wtf is happenning

#

This has never happened before

#

The scheduler just choking somewhere and refusing to run but on one cpu

#

or refusing to run at all

#

The scheduler lock is unlocked

#

The lapic timer isn't masked

#

IF in eflags is set on all cpus

flint idol Jan 20, 2024, 9:05 PM

#

flint idol The lapic timer isn't masked

The lapic timer is setup correctly

inland radish Jan 20, 2024, 9:07 PM

#

flint idol The scheduler just choking somewhere and refusing to run but on one cpu

I had a bug where sometimes the LAPIC timer would manage to interrupt "too early", so I had to add some code to my IRQ handler to re-request an interrupt in a couple hundred ticks, a very unclean solution but works

flint idol Jan 20, 2024, 9:15 PM

#

The timer interrupt isn't being called in the first place

flint idol Jan 20, 2024, 9:36 PM

#

I got into usermode!

#

and page faulted

flint idol Jan 20, 2024, 10:21 PM

#

I got further to the point where the syscall instruction was executed

#

Then page faulted in the syscall instruction handler

flint idol Jan 21, 2024, 12:57 AM

#

Idk how, but there's stack corruption coming from new

#

How fun!

#

But only in obos::syscalls::canAccessUserMemory

#

I think it's because of __builtin_return_address somehow

#

Nvm it's something else, what that is, I don't know

#

It's because of the TSS stack being overwritten by a page fault

#

and it seems as if the TSS stack is the same as the stack switched to in the syscall instruction handler

#

I use ISTs for page fault handlers btw

#

and the IST stack is always the same as RSP0

#

(or at least most of time it should be)

#

Maybe that's my issue

#

If I use the current CPU temporary stack that might fix something

#

But the scheduler uses that too, what if the scheduler is called, and the scheduler returns without preempting

#

thonk

#

Add another stack for syscalls? That seems wasteful though

flint idol Jan 21, 2024, 1:37 AM

#

Ok that fixed it

#

I got page table corruption

#

#

Obviously the phyisical address of a page table should never be greater than the amount of memory that can be physically addressed

astral mason Jan 21, 2024, 1:44 AM

#

Of course, because 5 thousand things must go wrong before something actually works.

flint idol Jan 21, 2024, 1:44 AM

#

Yeah of course

#

That's how it is in OBOS

#

I mean nothing a rebuild doesn't fix

astral mason Jan 21, 2024, 1:47 AM

#

When in doubt, rebuild it.

flint idol Jan 21, 2024, 1:47 AM

#

Yes.

#

I rebuilt it and that assertion didn't fail

#

Btw I made a new branch for userspace developement:
https://github.com/oberrow/obos/tree/userspace-work

GitHub

GitHub - oberrow/obos at userspace-work

I was bored... Contribute to oberrow/obos development by creating an account on GitHub.

astral mason Jan 21, 2024, 1:55 AM

#

Neat! I'll be sure to check it out!

flint idol Jan 21, 2024, 12:26 PM

#

sigh

#

I think I know what could've happened

#

A call of the function schedule with interrupts enabled

#

Which I was doing in ExitThread and ExitProcessImpl

#

to yield

#

Even though I should always call thread::callScheduler to yield

flint idol Jan 21, 2024, 1:22 PM

#

I fixed ExitThread and TerminateProcess (on the current process)

#

Which means I can get back to working on syscalls

#

; This file is released to the public domain.
; Example syscall code for oboskrnl on x86-64
; Compile with nasm.

[BITS 64]
[DEFAULT REL]

global _start

_start:
    push rbp
    mov rbp, rsp

;  Call ExitThread(0xDEADBEEF)
    mov eax, 0xDEADBEEF
    push 0
    push rax
    mov rax, 12
    mov rdi, rsp
    syscall
    add rsp, 16

    leave
    ret

#

That's the test program I made

#

and because cmake, there was a dummy.c file

#

I needed it for the link command to work properly and not use nasm as a linker

flint idol Jan 21, 2024, 4:27 PM

#

Because why not, the thread of the test program is getting corrupted

flint idol Jan 21, 2024, 4:42 PM

#

I'm removing the syscalls interface through interrupts

flint idol Jan 21, 2024, 4:58 PM

#

flint idol Because why not, the thread of the test program is getting corrupted

Upon further inspection, it isn't

#

Servicing hardware INT=0x20
   728: v=20 e=0000 i=0 cpl=3 IP=01c3:00000000004000bd pc=00000000004000bd SP=01bb:0000000000010fe8 env->regs[R_EAX]=0000000000000000
RAX=0000000000000000 RBX=0000000000000000 RCX=00000000004000bd RDX=fffffffff003dd70
RSI=0000000000000000 RDI=0000000000000000 RBP=0000000000010ff0 RSP=0000000000010fe8
R8 =0000000000000000 R9 =0000000000019000 R10=0000000000010fe8 R11=0000000000203246
R12=0000000000000000 R13=0000000000000000 R14=0000000000000000 R15=0000000000000000
RIP=00000000004000bd RFL=00203246 [---Z-P-] CPL=3 II=0 A20=1 SMM=0 HLT=0
ES =0000 0000000000000000 00000000 00001300
CS =01c3 0000000000000000 ffffffff 00a0fb00 DPL=3 CS64 [-RA]
SS =01bb 0000000000000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
DS =0000 0000000000000000 00000000 00001300
FS =0000 0000000000000000 00000000 00001300
GS =0000 fffffffff0002d68 00000000 00001300
LDT=0000 0000000000000000 0000ffff 00008200 DPL=0 LDT
TR =0028 fffffffff0002dd8 00000063 00408900 DPL=0 TSS64-avl
GDT=     fffffffff0002da0 00000037
IDT=     ffffffff80029900 00000fff
CR0=80010013 CR2=0000000000000000 CR3=000000003f732000 CR4=00310620
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
CCS=0000000000000044 CCD=0000000000018ff8 CCO=EFLAGS
EFER=0000000000000d01```

#

Those are the values I got in the stack frame for the thread

#

It GPFs while switching to this saved context

#

Obviously, cs and ss are garbage

#

But for it to preempt with garbage in cs and ss, the thread would've needed to run before in the first place

#

that could be because of the ldt

#

the limit of the ldt seems to be different while switching to the thread and when it was preempted

#

Even when I load the ldt with 0x0000 on all cpus it faults while switching to the context

vernal chasm Jan 21, 2024, 5:41 PM

#

flint idol ```txt Servicing hardware INT=0x20 728: v=20 e=0000 i=0 cpl=3 IP=01c3:0000000...

I also get that like a billion times kekw

#

Also how many segments descriptors do you have???

flint idol Jan 21, 2024, 6:06 PM

#

5 not including tss

#

With tss 6

flint idol Jan 21, 2024, 8:14 PM

#

The bug is with sysret

#

(of course on my side)

flint idol Jan 21, 2024, 9:10 PM

#

Ummm
So a thread has the status THREAD_STATUS_CAN_RUN | THREAD_STATUS_DEAD

#

meme

near linden Jan 21, 2024, 9:11 PM

#

flint idol But for it to preempt with garbage in cs and ss, the thread would've needed to r...

syscall maybe?

near linden Jan 21, 2024, 9:12 PM

#

flint idol The bug is with sysret

oh nevermind

flint idol Jan 21, 2024, 9:13 PM

#

flint idol Ummm So a thread has the status `THREAD_STATUS_CAN_RUN | THREAD_STATUS_DEAD`

It's because in the scheduler if a thread sets it's status to THREAD_STATUS_DEAD and calls the scheduler, the scheduler would set THREAD_STATUS_CAN_RUN, like it always does. I added a check, it should work properly now.

#

Everytime I make a change to the scheduler, everything seems to hang, probably because of incremental building. I'll try a rebuild

#

Oh nvm it's a bug

#

I only set THREAD_STATUS_CAN_RUN if the thread is dead, not if it's not dead

#

I added a test thread to the test program to test the thread syscalls, they seem to work

#

Time to test file syscalls

flint idol Jan 21, 2024, 11:23 PM

#

They work

#

Time to add console syscalls

flint idol Jan 21, 2024, 11:50 PM

#

Done, now time to test them

flint idol Jan 22, 2024, 12:19 AM

#

That failed terribly...

#

I suspect it has something to do with the fact I allocated a new console

#

#

I made the test program load a splash message from disk and print it

#

It also clears the terminal

#

I now need to implement disk access syscalls

#

then process manipulation syscalls

#

then a ps/2 keyboard driver

#

then a way to interface with input devices

#

(possibly with the VFS)

flint idol Jan 22, 2024, 12:35 AM

#

flint idol I now need to implement disk access syscalls

actually no vmm syscalls then disk access

#

I added a Doxygen file for syscalls

#

Every syscall so far is documented with XML comments

#

and doxygen supports those

flint idol Jan 22, 2024, 1:47 AM

#

Writing the test program in assembly is starting to get annoying

#

So I'll just write it in C

flint idol Jan 22, 2024, 2:33 AM

#

I'm going to make the test program print the contents of a file to the console, with the buffer being allocated with the vmm syscalls, in a newly created thread

#

To test many things at once

flint idol Jan 24, 2024, 1:20 AM

#

I might take a break for a few days (or just today)

chilly osprey Jan 24, 2024, 1:21 AM

#

flint idol Time to add console syscalls

console baked into the kernel nooo

flint idol Jan 24, 2024, 1:21 AM

#

Don't worry I'll (eventually) provide a way to mmap a framebuffer

chilly osprey Jan 24, 2024, 1:21 AM

#

good

flint idol Jan 24, 2024, 1:22 AM

#

Once I get to porting a desktop environment (or making my own), I'll provide that

chilly osprey Jan 24, 2024, 1:24 AM

#

i would personally worry about that considerably later

flint idol Jan 24, 2024, 1:24 AM

#

Yeah I know

#

There are a lot more stuff I have to do

chilly osprey Jan 24, 2024, 1:24 AM

#

that is an exciting milestone of course

#

but there are plenty of others before that(in my opinion)

flint idol Jan 24, 2024, 1:25 AM

#

In my opinion if the only way to shutdown your os is through holding the power button (so no way to do it in the OS), then you're not close to being ready

chilly osprey Jan 24, 2024, 1:28 AM

#

thats not really something that is required for a lot of things

flint idol Jan 24, 2024, 1:28 AM

#

Just saying

chilly osprey Jan 24, 2024, 1:30 AM

#

i think even toaruos doesnt even have a shutdown

#

i may be wrong there though

flint idol Jan 24, 2024, 1:31 AM

#

isn't that klange's os

#

~~we'll ask~~

#

@ klange

chilly osprey Jan 24, 2024, 1:31 AM

#

i would ask but i think its quite an awkward time in japan at the moment

#

10:30am, maybe not

#

but still, no need to bother him

flint idol Jan 24, 2024, 1:32 AM

#

flint idol @ klange

it'd be funny if I accidentally pinged him when I sent this message

#

or her

#

or them

rapid pulsar Jan 24, 2024, 2:00 AM

#

Shutdown is one of those funky ACPI things, and I don't have anything beyond the tables needed for SMP.

#

But if you're sitting on LAI or acpica...

flint idol Jan 24, 2024, 2:01 AM

#

How did you detect me asking that question

lean glen Jan 24, 2024, 2:10 AM

#

https://tenor.com/view/sauron-lotr-lord-of-the-rings-mordor-the-eye-gif-5619682

Tenor

rapid pulsar Jan 24, 2024, 2:13 AM

#

I never leave a channel unread, and I've posted here so this thread shows up in my sidebar.

flint idol Jan 25, 2024, 1:58 AM

#

I think I might redo my code for looking for free pages

#

My current algorithm is, well, less than optimal

#

const uintptr_t base = isUserProcess ? USER_BASE : KERNEL_BASE;
const uintptr_t limit = isUserProcess ? USER_LIMIT : KERNEL_LIMIT;
uintptr_t addr = base;
for (; addr < limit; addr += 0x1000)
    if (CanAllocatePages((void*)addr, nPages, pageMap))
        break;```

weary hound Jan 25, 2024, 2:01 AM

#

if this is for allocating virtual address space, you could hold the free areas and used areas in lists/trees/whatever

flint idol Jan 25, 2024, 2:01 AM

#

It's for the VMM yeah

#

Why make my own structures when the page tables can be iterated without extra overhead

#

I think I'll have the algorithm loop over all mapped addresses, and find a region big enough to the hold the amount pages needed

weary hound Jan 25, 2024, 2:03 AM

#

because in the list/tree you can hold spans larger than a single page, which would certainly be faster if you're searching for a large enough region

#

and in case you ever page out pages the page table entries vanish (or at least have their present bit unset, and in this case you need a place to store extra bookkeeping information anyway)

vernal chasm Jan 25, 2024, 3:23 AM

#

I did that in conjunction with a lut based freelist

flint idol Jan 26, 2024, 2:52 AM

#

Idk why, but ExitThread now triple faults...

#

For some reason, the thread context's kernel stack is nullptr

#

I'll debug it tomorrow

flint idol Jan 26, 2024, 1:09 PM

#

It was a bug with the new algorithm to find free pages

#

I needed to use nPages + 1 instead of nPages when checking if there's enough space between two mapped pages

flint idol Jan 26, 2024, 2:31 PM

#

I should add a framebuffer lock

#

This is why:

#

I added a framebuffer lock, at least the text doesn't get cut off mid-swap

#

One thing I could do is wait for the kernel boot thread to exit before doing anything with the console

#

I think I found a possible bug with SMP

#

or with something idk what

#

But with 2 cpus, the init program refuses to run without page faulting while setting the colour of the console

#

With 1 cpu, it well:

#

The scheduler does not like one cpu

#

or something

#

idk at this point

#

I'll try 8 cpus

#

Everything shits itself with anything other than 4 cpus it seems

#

Weirdly with 16 cpus it lags more

#

But it works

#

64 cpus, it triple faults

#

Nvm

#

It seems as if the more cpus I add the more the kernel lags

#

I think it's the scheduler lock

#

~~Solution: remove the scheduler lock~~

#

Hmmm

#

I'll try that (it will most definitely shit itself)

#

Wait what

#

nvm

#

It was one lucky run

#

128 cores lol

#

with no scheduler lock

#

what could go wrong

#

and with 64-bit affinity

#

it's interesting however, how one cpu runs faster than 64

#

3 cpus

#

I didn't try 32 cores

#

< 4 Cores: Shits itself
4,8,16 Cores: Works
> 16: Shits itself
8 Cores: Optimal performance.

#

I fixed the bug with one core

#

I need to check if the fix works with 2 cores

flint idol Jan 26, 2024, 3:13 PM

#

flint idol I fixed the bug with one core

*accidentally fixed

flint idol Jan 26, 2024, 3:13 PM

#

flint idol ``` < 4 Cores: Shits itself 4,8,16 Cores: Works > 16: Shits itself 8 Cores: Opti...

1,2,4,8,16 Cores: Works
> 16 Cores: Rather shits itself, or the performance sucks
> 128 Cores: No effect, affinity is a __uint128_t
8 Cores: Optimal performance.

#

I changed affinity to be a 128-bit integer

#

Let me see does the scheduler shit itself

#

No

#

How about with 128 cpus

#

It indeed does

#

Well it page faults

#

After a restart of qemu it did

#

There is no way

#

It just worked

#

#

Anyway that's enough playing with SMP

#

Time to commit the new code

inland radish Jan 26, 2024, 4:30 PM

#

flint idol Time to commit the new code

Should have committed already. If not in master/main, then in a new branch.

flint idol Jan 26, 2024, 8:36 PM

#

I just made DriveHandle syscalls

#

Too lazy to test them

#

I'm going to merge userspace-work with master

flint idol Jan 26, 2024, 9:04 PM

#

I'm porting the ps/2 keyboard driver from my previous kernel to my current kernel

#

Time to implement the IOAPIC

flint idol Jan 26, 2024, 10:58 PM

#

Done, now time to test it

#

First I want to add a syscall to load a driver

flint idol Jan 27, 2024, 1:29 AM

#

Ok time to test this on real hardware

#

I completely forgot to do that while debugging syscalls

flint idol Jan 27, 2024, 2:54 AM

#

Of course there's page table corruption

#

How do I mess up this badly?

#

I'll debug it tomorrow

ornate ginkgo Jan 27, 2024, 4:30 AM

#

too lazy to test, merging with master

flint idol Jan 27, 2024, 1:20 PM

#

But that was something different

shadow steeple Jan 27, 2024, 1:51 PM

#

flint idol This is why:

0xdeadbeef exitcode! This is the why we should use EXIT_SUCCESS and EXIT_FAILURE constants in a C program even "Hello, World!". 🙂

flint idol Jan 27, 2024, 1:52 PM

#

I made it exit with that code

#

A way of testing thread handles even after the underlying thread died

shadow steeple Jan 27, 2024, 1:52 PM

#

Good project I think.

flint idol Jan 27, 2024, 1:52 PM

#

The last thread handle to get closed frees the thread object

#

On a side note I did get the ps2 keyboard driver to work with the IOAPIC

#

Which means after some bug fixes and adding a virtual stdin file, I can make a terminal

flint idol Jan 27, 2024, 5:52 PM

#

flint idol Of course there's page table corruption

It was a missing memset(pageMap, 0, 4096)

#

I think

#

That was the problem

flint idol Jan 27, 2024, 6:29 PM

#

The PS/2 Keyboard Driver works on real hardware

#

I'm going to add a way to read bytes from the keyboard through the VFS

#

But before that, I'm going to commit the new code

vernal chasm Jan 27, 2024, 7:11 PM

#

flint idol But before that, I'm going to commit the new code

Guessing commit broke :>

flint idol Jan 27, 2024, 7:11 PM

#

wdym

vernal chasm Jan 27, 2024, 7:11 PM

#

Yes kekw

flint idol Jan 27, 2024, 7:12 PM

#

Anyway I'm implementing extended scan codes in my ps/2 keyboard driver

vernal chasm Jan 27, 2024, 7:12 PM

#

But where usb driver

flint idol Jan 27, 2024, 7:12 PM

#

eventually

vernal chasm Jan 27, 2024, 7:12 PM

#

Oof

flint idol Jan 27, 2024, 8:47 PM

#

I'm done implementing extended scancodes

#

Now how should I implement stdin

#

It could be a per-process object, but then how do I know which process to give keyboard input to

#

or what if I don't implement stdin

#

and add syscalls to read from input devices and make "virtual devices"

#

and virtual devices could be some sort of pipe-type structure

#

So the process opens a handle to a pipe which it reads from and the keyboard driver writes to

#

Then it's the window manager's problem to decide which process gets the keyboard input

#

To do that I'll add a syscall to make the pipe drop all data written

#

and one to stop dropping data written

#

I'll also implement some form of handle inheritance

#

So a process can pass on a pipe handle to any child processes

flint idol Jan 27, 2024, 10:42 PM

#

Got a 'bit' off track

#

I'm now implementing uACPI

#

https://github.com/UltraOS/uACPI

flint idol Jan 28, 2024, 7:39 PM

#

I can now shutdown

#

Exciting I know

limber wave Jan 28, 2024, 8:20 PM

#

woaw

flint idol Jan 30, 2024, 12:10 AM

#

I'm going to play around with ACPI sleep states on real hardware. What could possibly go wrong

#

I'll try S3 first

#

or well second

#

I tried S4 and my laptop didn't support that

#

No S3 which is surprising

#

I'll try S2

#

I have no idea if this is working

#

No lights go off

#

nor does the monitor

#

I'll try on the other test computer

#

This computer supports S4

#

Based off the bios settings

#

S4 worked

#

It shutdown, but post seemed to be a lot faster

#

Because well S5 is the shutdown state

#

S4 is hibernate, as windows calls it

ashen goblet Jan 30, 2024, 12:56 AM

#

do you do any cleaning up on shutdown, like memory deallocation

#

or do you just shut down halfmemeright

flint idol Jan 30, 2024, 1:49 AM

#

Just shutdown

#

For now

#

I'm planning to add a callback for drivers to cleanup before shutdown

#

Because an NIC driver might want to stop transmitting data before shutdown

#

However before I do that I want to implement automatic driver loading for ACPI

#

I got the code, just I need to test it

#

I was the first adopter of uACPI, before even the creator implemented it

#

I helped him test it on real hardware

#

He found four bugs with LAI while making his AML interpreter, and even a bug with ACPICA

flint idol Jan 30, 2024, 3:24 AM

#

I can now detect devices on the ACPI namespace and load the appropriate driver for it

#

Which includes the PS/2 Keyboard Driver (I know I could've used the FADT table, but this is cooler)

#

Uhhh

#

Why does the kernel hang when I type

#

while booting

#

I do output the characters to the terminal

#

That might be why, probably a deadlock on the kernel console

#

Anyway it works on my laptop

flint idol Jan 30, 2024, 3:24 PM

#

I forgot to mention this but now I've hooked the keyboard interrupt on panic to rather shut down or reboot the computer (depending on whether uacpi is initialized or not)

flint idol Jan 30, 2024, 10:48 PM

#

Enough playing with acpi, time to get back to keyboard input

flint idol Jan 31, 2024, 1:37 AM

#

I've decided how keyboard input will work on the user side

#

You make a file handle to "Kn", with n being the keyboard id registered by the input driver

#

So to echo every character printed, you'd just do this:

uintptr_t keyboardHandle = MakeFileHandle();
OpenFile(keyboardHandle, "K0", 0);
char ch[2] = {};
while (1)
{
    while (Eof(keyboardHandle));
    ReadFile(keyboardHandle, ch, 1, false);
    ConsoleOutput(ch);
}```

#

But heap corruption happens after a few characters

#

Function kmalloc, File C:/Code/obos/src/oboskrnl/allocators/liballoc.cpp, Line 259: Assertion failed, "currentPageBlock->highestBlock->magic == MEMBLOCK_MAGIC". Kernel heap corruption detected for block 0xfffffffff009fa30, allocAddr: 0xfffffffff009faa0, sizeBlock: 0x0000000000000010!
Stack trace:
        0xffffffff8001a43b: _ZN4obos6logger13panicVariadicEPvPKcP13__va_list_tag+148
        0xffffffff8001a311: External Code
        0xffffffff8001ab13: kmalloc+890
        0xffffffff8001b33f: _Znam+24
        0xffffffff8001224e: _ZN4obos8syscalls19canAccessUserMemoryEPKvmb+180
        0xffffffff80012501: _ZN4obos8syscalls24FileHandleSyscallHandlerEmPv+439
        0xffffffff80002fb9: External Code
        0x00000000004000c5: External Code
        0x0000000000400170: External Code
        0x000000000040079d: External Code
        0x00000000004007e9: External Code
Press any key to shutdown...```

#

In this case, it's probably the highest block being a dead block

#

So magic == MEMBLOCK_DEAD

#

The structure is purposefully kept intact so the allocator can reallocate free blocks

#

In this case it's faulting from the ReadFile syscall

#

Based on the stack trace

#

I'll see if this was just the kernel being dumb or if it's an actually bug

#

If not

#

I'll make a small terminal

#

When I do sendkey ctrl-alt-delete in the qemu monitor it reports heap corruption after a few tries

#

But with normal input it never reports heap corruption

#

For some reason the alt key is registered as the 0 character

ornate ginkgo Jan 31, 2024, 1:46 AM

#

something to do with how you handle modifiers?

flint idol Jan 31, 2024, 1:46 AM

#

Probably

flint idol Jan 31, 2024, 2:48 AM

#

I got heap corruption, clicked a key so the computer could shut down, then it page faulted while allocating an object

#

My kernel is very stable lol

flint idol Jan 31, 2024, 2:49 AM

#

flint idol I got heap corruption, clicked a key so the computer could shut down, then it pa...

or while allocating it detected heap corruption again

flint idol Jan 31, 2024, 7:44 PM

#

The bug seems to be because of the keyboard buffer not being reallocated properly

#

I think that because 16 bytes is the pointer and size alignment and the 17th byte typed reports heap corruption

#

Because the corruption is referring to highestBlock and krealloc assumes that if a block being reallocated is the highest block and there is enough space remaining in the page block the block can expanded, I think there is a bug in kfree when it sets the new highest block

#

So that the highest block is not actually the highest block

#

and by highest block I mean the block that has the highest address

flint idol Jan 31, 2024, 10:25 PM

#

I'm making what I think is a slab allocator

flint idol Feb 1, 2024, 12:20 AM

#

I just finished making and testing the slab allocator

#

It works

#

Or at least I think it's a slab allocator, idk

#

It has a free list and a list of allocated blocks per type of object that it allocates. To allocate it removes an entry from the free list, adds it to the allocated list, then returns the node + sizeof(*node). To free it does the opposite, it adds the node to the free list and removes it from the allocated list.

#

The objects it can allocate are below:

enum class ObjectTypes
{
    Invalid,
    Thread,
    Process,
    Mutex,
    MountPoint,
    DirectoryEntry,
    PartitionEntry,
    DriveEntry,
    InputDevice,
    DriverIdentity,
};```

#

and so I didn't have to change much code, I overloaded the new/delete operators for each one of these classes

#

For some reason I had to make placement new/delete or the compiler would complain

flint idol Feb 1, 2024, 11:54 PM

#

Ok what the hell

#

How did I just fix the bug without doing anything

#

Anyway I'm going to make a embedded controller driver

flint idol Feb 2, 2024, 12:13 AM

#

*Eventually I will make a embedded controller driver

#

Now I'm going to write a basic shell

#

In user mode

#

I need an allocator so I'll just copy and paste liballoc from the kernel

flint idol Feb 2, 2024, 12:55 AM

#

You know it's a lot easier to code a shell with a libc

#

For future reference

#

#

IT WORKS

#

I'm going to make a "cat" command

#

#

I got cat working

#

There's a bug with the getline function

#

I'm going to fix it quickly

flint idol Feb 2, 2024, 1:15 AM

#

flint idol There's a bug with the getline function

When you backspace and size == 0, it didn't check for size being zero and tried to reallocate size - 1 bytes, which is -1, which is actually 0xffff'ffff'ffff'ffff because size is a size_t

flint idol Feb 2, 2024, 12:14 PM

#

I think I've had a race condition this entire time with thread handles, all because I forgot to increment obj->references in the thread handle's OpenThread and CreateThread functions. That caused ExitThread to free the thread even with open handles to that thread.

#

Also signals don't work

#

I need to fix those

flint idol Feb 3, 2024, 12:21 AM

#

I added syscall to shutdown and reboot

#

and I added commands in my shell for shutting down and rebooting

#

All the shutdown command does is EnterSleepState(5)

unkempt pawn Feb 3, 2024, 1:30 AM

#

make sure that unprivileged code cant use shutdown and reboot

flint idol Feb 3, 2024, 1:31 AM

#

Well for that, I'd need a privellege system for user space, which I don't have

#

Assuming you mean like only root can call the shutdown syscall, or it will fail

unkempt pawn Feb 3, 2024, 1:40 AM

#

You should implement a privellege system

flint idol Feb 3, 2024, 1:40 AM

#

In case anyone cares, when you open a file with my kernel, it expects an absolute path, it can't even have stuff like ../ or ./ or OpenFile will fail. Also no working directories, so the path has to refer to a mount point

flint idol Feb 3, 2024, 1:41 AM

#

unkempt pawn You should implement a privellege system

I've been thinking how, and I thought of making a bytecode format so that the privillege system is completely customizable

#

It's not final though

#

on what I'll do

flint idol Feb 3, 2024, 1:43 AM

#

flint idol I've been thinking how, and I thought of making a bytecode format so that the pr...

the bytecode would be set once, say by the init program, then it can't be modified except with appropriate permissions

#

that are given by the bytecode

inland radish Feb 3, 2024, 5:34 AM

#

flint idol In case anyone cares, when you open a file with my kernel, it expects an absolut...

You can just keep a reference to the CWD and start path lookup from there if your path doesn't start with a slash

ornate ginkgo Feb 3, 2024, 11:51 AM

#

flint idol I've been thinking how, and I thought of making a bytecode format so that the pr...

use AML meme

flint idol Feb 3, 2024, 12:01 PM

#

I mean there is an AML interpreter implemented in the kernel

#

Not my AML interpreter of course

flint idol Feb 3, 2024, 7:09 PM

#

Error code 0: OBOS_ERROR_SUCESS
I just love it when I fail sucessfully

#

I implemented directory iterator syscalls

#

Wait it was because I forgot to pass in the second format specifier

#

GetLastError: 19
Error 19 is OBOS_ERROR_VFS_FILE_NOT_FOUND

#

1:/ is a fat32 mount point, and 0:/ is the initrd mount point

#

and both fail when there is a subdirectory

flint idol Feb 3, 2024, 7:34 PM

#

Don't you just love it when the allocator randomly breaks after changing precisely zero things with it

#

Ok this isn't very safe code

while (current != nullptr && current < currentPageBlock->lastBlock)
{
    OBOS_ASSERTP(GET_FUNC_ADDR(current) > 0xffff'8000'0000'0000, "Kernel heap corruption detected.");
    if (current->magic != MEMBLOCK_DEAD)
    {
        current = current->next;
        continue;
    }
    if (current->size >= (amountNeeded - sizeof(memBlock)))
    {
        block = current;
        break;
    }
    current = current->next;
}```

#

Why didn't I make a free list here?

#

Why'd I make it loop through blocks which are likely overwritten

#

Ok this code sucks

#

That loop was so a free region could be found

#

The heap will be a lot more fragemented, but at least no more heap corruption

flint idol Feb 3, 2024, 8:33 PM

#

My kernel is so fast that clicking a key sometimes brings two keys because of how little delay there is, rather that or it's a quirk with usb->ps2 keyboard emulation

flint idol Feb 4, 2024, 7:19 PM

#

Scheduler bug pain

#

Ok how the fuck does this thread get scheduled when it is clearly blocked

#

Why is an idle thread blocked in the first place

#

Ohhhhhhhh

#

An idle thread received a keyboard interrupt which locked a lock

#

Which was locked, so Mutex::Lock blocked the thread and called the scheduler

#

But the scheduler always schedules the cpu's idle thread if there are no other tasks, which there aren't

#

Possible fixes:
Start a thread to handle interrupts
Make sure no one blocks the current thread if it is the idle thread

#

The first has the con that some interrupts might need to be tended to immediately

#

The second one has the con that everywhere there is a function that blocks the current thread, it needs to check if the current thread is the idle thread

#

One other solution is in the scheduler, it checks if the idle thread is blocked if the idle thread yields execution, and if so it just runs the block callback in a loop waiting for it to become unblocked

flint idol Feb 4, 2024, 8:56 PM

#

I want a microcontroller architecture to port OBOS to, I think Xtensa (the stuff that powers ESP32) is an interesting choice. The only requirements are realistically paging, a minimum of 32 bits for the processor (32+ bit addresses and 32+ bit gpr registers), and interrupts. User mode is optional, but nice

#

The core-kernel actually knows nothing about user mode or syscalls

#

By core-kernel I mean the kernel not including arch-specific stuff

flint idol Feb 4, 2024, 8:58 PM

#

flint idol The core-kernel actually knows nothing about user mode or syscalls

not including the isUsermode fields in process::Process and process::CreateProcess

flint idol Feb 4, 2024, 11:29 PM

#

I'm porting mlibc because I think it's time

#

First I got to find out how to change what meson is compiling for

#

specifically changing host_machine.system()

steep ravine Feb 5, 2024, 4:24 PM

#

meson cross file

#

look at any project using mlibc

flint idol Feb 5, 2024, 5:05 PM

#

I found it out

#

I just need to figure out gcc os specific tool chains

flint idol Feb 8, 2024, 2:01 AM

#

#

x86_64-elf-obos-gcc -dumpmachine returns x86_64-elf-obos

#

Now I'll make a test program

#

It'll just include a libc header to test if that works

flint idol Feb 8, 2024, 2:04 AM

#

flint idol Now I'll make a test program

*file

#

I haven't ported binutils

#

I didn't add a sysroot

#

I need to fix that

flint idol Feb 8, 2024, 2:29 AM

#

# !/usr/bin/bash

export TARGET=x86_64-elf-obos
export PREFIX=$HOME/x86_64-elf-obos/

./configure --target=$TARGET --prefix=$PREFIX --disable-nls --enable-languages=c,c++ --disable-multilib
make -j16 all-gcc

To build gcc with x86_64-elf-obos as the target.
You should just be able to clone a gcc mirror and git am these changes then build gcc with the correct target:

📎 0002-Fix-the-config-files.patch 📎 0001-Add-untested-configuration-for-OBOS.patch

#

or maybe just the second patch file

#

idk

#

Anyway tomorrow I'll probably port binutils

#

Then get back to porting mlibc

#

The macro __obos__ is defined

#

So I know that works

sharp pike Feb 8, 2024, 4:27 AM

#

uh....

#

it's not supposed to be elf-obos

#

just x86_64-obos

#

the elf part there is just treated like the machine name thing aka ignored and useless

weary hound Feb 8, 2024, 8:45 AM

#

when configuring your gcc for your target you probably want to point it at your sysroot

#

with --with-sysroot=/path/to/sysroot

flint idol Feb 8, 2024, 10:51 AM

#

Yeah I said I'd do that

flint idol Feb 8, 2024, 10:52 AM

#

sharp pike the elf part there is just treated like the machine name thing aka ignored and u...

So would I be able to just do target as obos

flint idol Feb 8, 2024, 11:27 AM

#

Oh nvm

flint idol Feb 8, 2024, 9:31 PM

#

I ported binutils

flint idol Feb 9, 2024, 12:53 AM

#

I've compiled gcc like five times in the same week

sharp pike Feb 9, 2024, 1:29 AM

#

only?

flint idol Feb 9, 2024, 1:29 AM

#

That was only an estimate

#

Not including unsucessful builds

flint idol Feb 9, 2024, 2:27 AM

#

I think tomorrow I'm going to start implementing swap and mapping files.

flint idol Feb 10, 2024, 6:56 PM

#

I got memory mapped files to work

#

Swap shouldn't be hard, as I already have half the work done

flint idol Feb 10, 2024, 9:20 PM

#

I'm rewriting the kernel

#

ashen goblet Feb 10, 2024, 9:22 PM

#

flint idol I'm rewriting the kernel

That could be fun tho

#

gl

flint idol Feb 10, 2024, 9:23 PM

#

This time I'm designing it instead of "Random shit go!"

ashen goblet Feb 10, 2024, 9:23 PM

#

I feel ya

flint idol Feb 10, 2024, 9:23 PM

#

Or at least having a plan

limber wave Feb 11, 2024, 1:15 PM

#

sometimes random shit is fun

#

but i guess depends on goals

flint idol Feb 11, 2024, 1:15 PM

#

limber wave sometimes random shit is fun

Not when you want to port software

limber wave Feb 11, 2024, 1:15 PM

#

oh yea true

flint idol Feb 11, 2024, 1:15 PM

#

Anyway the screenshot I sent have the other reasons for me rewriting

#

I'm not going to follow posix, but at least it will be planned

limber wave Feb 11, 2024, 1:16 PM

#

good luck

flint idol Feb 11, 2024, 1:16 PM

#

Ty

limber wave Feb 11, 2024, 1:16 PM

#

how much loc are you rewriting?

flint idol Feb 11, 2024, 1:17 PM

#

Everything

limber wave Feb 11, 2024, 1:18 PM

#

oh wait you literally deleted the entire repo

flint idol Feb 11, 2024, 1:18 PM

#

Yeah

limber wave Feb 11, 2024, 1:18 PM

#

damn

#

#

25k LOC :P

flint idol Feb 11, 2024, 1:19 PM

#

I'm assuming that was for my previous kernel

limber wave Feb 11, 2024, 1:19 PM

#

yea makes sense why youd want to delete it all

limber wave Feb 11, 2024, 1:19 PM

#

flint idol I'm assuming that was for my previous kernel

yes

flint idol Feb 11, 2024, 1:20 PM

#

Too bad my gcc configs going unused for some time

#

I'm making the build system download dependencies this time

#

Instead of saying: Here's a dependency, clone it yourself

vernal chasm Feb 11, 2024, 2:40 PM

#

flint idol I'm not going to follow posix, but at least it will be planned

PogW

flint idol Feb 11, 2024, 5:54 PM

#

I got the GDT pushed

#

Nvm

#

ok now I have the GDT pushed

flint idol Feb 13, 2024, 1:05 AM

#

I got the IDT working

#

Time to work on the IOAPIC, APIC, and interrupt priorities

flint idol Feb 13, 2024, 11:51 PM

#

I implemented those, time to make the PMM

devout niche Feb 13, 2024, 11:56 PM

#

flint idol I got the IDT working

wait a minute, you had memory mapped files and now you're back to GDTs and IDTs?

flint idol Feb 13, 2024, 11:57 PM

#

#1141057599584878645 message

flint idol Feb 14, 2024, 12:31 AM

#

I finished implementing the PMM (Definitely didn't copy+paste my previous kernel's PMM)

flint idol Feb 17, 2024, 12:51 AM

#

I think I might take a bit more time on the VMM this time

#

Then I'm working on the scheduler

flint idol Feb 18, 2024, 1:41 AM

#

I got mapping pages working

#

I also make my own page tables upon initialization of the VMM

#

Instead of keeping limine's and just copying over the entries

flint idol Feb 18, 2024, 1:42 AM

#

flint idol I got mapping pages working

With 4kib and 2mib page sizes

#

Which meant I also added an option to the PMM to allocate on a 2 mib boundary

vernal chasm Feb 19, 2024, 11:28 PM

#

flint idol With 4kib and 2mib page sizes

Now do 1GiB pages :>

ashen goblet Feb 20, 2024, 3:08 PM

#

oberrow, sorry if I annoy you, but since you said something about wanting to take a better approach at designing your new vfs, I wanted to ask if you had found any helpful resources along the way that you can recommend, or some simple but good implementations that you use as ref

flint idol Feb 20, 2024, 3:17 PM

#

No I didn't get to the vfs yet

#

still on vmm

#

But check the pinned messages in #filesystems

ashen goblet Feb 20, 2024, 3:21 PM

#

k

#

What vmm implementation are you going for?

flint idol Feb 20, 2024, 3:22 PM

#

Idk yet

#

Lol

vernal chasm Feb 21, 2024, 4:59 AM

#

Yes vmm takes time to architect

flint idol Feb 21, 2024, 11:06 AM

#

Yes.

flint idol Feb 24, 2024, 6:06 PM

#

I had to take a break because of homework

#

But today I will work on vmm design

#

and stuff

ashen goblet Feb 24, 2024, 9:04 PM

#

flint idol I had to take a break because of homework

feel you

flint idol Mar 1, 2024, 12:20 PM

#

Project isn't dead, it's just there isn't much to say about making a slab allocator except that it's boring

#

I need it for allocating some stuff in the VMM, so I use some "raw" functions to allocate and free virtual pages.

#

More info on what I did is in the latest commit message (commit 84c7531e4a6b6fc03e3189a31ce829e5eb06bdfc)

flint idol Mar 3, 2024, 7:29 PM

#

I'm done the slab allocator, and I'm almost done the VMM

#

I just need to implement un map and demand paging

#

(for now)

#

I'll eventually implement memory being backed by files

#

But for that I'd need a concept of files

flint idol Mar 3, 2024, 11:03 PM

#

I've implemented demand paging

#

Time to implement un map

#

I forgot that SetProtection and GetPageDescriptor existed

#

I need to implement those aswell

#

I estimate the VMM will be done within a couple days, assuming I have time

chilly osprey Mar 4, 2024, 9:17 AM

#

flint idol I estimate the VMM will be done within a couple days, assuming I have time

what do you consider "done"

flint idol Mar 4, 2024, 11:38 AM

#

All functions I plan to implement are implemented

flint idol Mar 12, 2024, 5:18 PM

#

I've implemented all functions in the VMM and have (finally) merged the VMM branch with master

#

Time to work on the scheduler

#

However, I first need SMP

#

I plan on making the kernel SMP-enabled from the beginning

real pecan Mar 12, 2024, 7:48 PM

#

flint idol Time to work on the scheduler

Delayed work queues pls

flint idol Mar 12, 2024, 7:48 PM

#

Like in interrupt handlers

real pecan Mar 12, 2024, 7:49 PM

#

Yeah like an ability to schedule deferred work in an interrupt handler

flint idol Mar 12, 2024, 7:49 PM

#

Yeah I was planning to

#

My previous kernel didn't

real pecan Mar 12, 2024, 7:50 PM

#

This is basically mandatory if u wanna be able to handle ACPI events

real pecan Mar 12, 2024, 8:05 PM

#

have a bunch of new API for this

flint idol Mar 12, 2024, 8:18 PM

#

What is GPE

#

@real pecan

real pecan Mar 12, 2024, 8:21 PM

#

A general purpose event

flint idol Mar 12, 2024, 8:21 PM

#

for some reason I thought it was general protection exception

real pecan Mar 12, 2024, 8:22 PM

#

These usually have a matching "method" that must be executed by the interpreter whenever hardware requests it

#

e.g GPE 5 will most likely have a \_GPE._L05 method

#

these events do all sorts of thing

#

from power management to hotplug notification

flint idol Mar 12, 2024, 8:23 PM

#

also the LAPIC has a register called lvtThermalSensor, does that have anything to do with ACPI

real pecan Mar 12, 2024, 8:23 PM

#

nah i dont think so

real pecan Mar 12, 2024, 8:24 PM

#

real pecan A general purpose event

this is as opposed to a fixed event

#

fixed events are events handled by the kernel and not firmware

#

aka power button press, sleep button press

#

stuff like that

#

Oh and GPEs are used to wake the system from shallow sleep states

#

during init you enumerate devices that declare wake capability and configure them

flint idol Mar 12, 2024, 8:30 PM

#

What if you only wanted some devices to wake

#

Do you just not configure them

real pecan Mar 12, 2024, 8:30 PM

#

by configure i meant "configure for wake"

#

you explicitly tell it to wake you

#

device can declare something like:

can wake the system, deepest supported sleep state _S3
the deepest state I can be in to still wake the system D3
dependent power resources that must be on for this device to be able to wake the system
index of the GPE that will be generated on wake

real pecan Mar 12, 2024, 8:33 PM

#

flint idol What if you only wanted some devices to wake

basically, u dont need any device-specific drivers to enable wake

#

everything is handled by methods in acpi

flint idol Mar 13, 2024, 3:33 PM

#

For future reference, making a SMP trampoline is a lot easier when you have labels defining where all shared variables are. It's also easier to just jump from real mode to long mode instead of from real mode->protected mode->long mode

#

and if you're making one, make sure to set the PAT msr

flint idol Mar 14, 2024, 11:29 AM

#

Don't you just love debugging a sorting algorithm while debugging your vmm while debugging SMP

inland radish Mar 14, 2024, 11:39 AM

#

flint idol Don't you just love debugging a sorting algorithm while debugging your vmm while...

Pull the sorting algorithm out of the kernel and debug it with asan/ubsan 😉

flint idol Mar 14, 2024, 11:45 AM

#

Nvm I fixed it

#

I got SMP working

#

I think

#

Except for some reason qemu is dying

#

I put a spinlock on the logging function, printf, and on the console, and yet this mess happens

#

bool SpinLock::Lock()
{
    uint8_t oldIRQL = 0;
    RaiseIRQL(0xF, &oldIRQL);
    const bool excepted = false;
    while (__atomic_load_n(&m_locked, __ATOMIC_SEQ_CST));
    while (__atomic_compare_exchange_n(&m_locked, (bool*)&excepted, true, false, __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST));
    LowerIRQL(oldIRQL);
    return m_locked;
}```

inland radish Mar 14, 2024, 12:01 PM

#

flint idol ```c++ bool SpinLock::Lock() { uint8_t oldIRQL = 0; RaiseIRQL(0xF, &oldI...

BTW you usually want to raise IPL when acquiring the spinlock to disable scheduling on the core and lower it back to the previous level when releasing

#

It avoids certain classes of deadlock involving the scheduler

#

Plus of course it doesn't force other CPUs to spin on the lock until the owner releases it when the owner isn't even running

flint idol Mar 14, 2024, 12:04 PM

#

If I do that it'd mean that locks need to be held for a short amount of time, or the system would deadlock

#

But I should be doing that with locks anyway.

inland radish Mar 14, 2024, 12:08 PM

#

flint idol If I do that it'd mean that locks need to be held for a short amount of time, or...

Yeah usually you ought to keep locks acquired for short amounts of time

flint idol Mar 14, 2024, 12:16 PM

#

#

I think that somehow fixed it

flint idol Mar 14, 2024, 12:53 PM

#

yeah I'm switching back to hyper, the bootloader I initially used for the rewrite

ashen goblet Mar 14, 2024, 12:53 PM

#

No limine?

flint idol Mar 14, 2024, 12:54 PM

#

I've run into problems with limine regarding setting up my own page tables

flint idol Mar 14, 2024, 12:54 PM

#

ashen goblet No limine?

Yeah

#

Good thing I have the git history so it'll be easy enough

ashen goblet Mar 14, 2024, 12:54 PM

#

flint idol I've run into problems with limine regarding setting up my own page tables

🧐

#

What happened

flint idol Mar 14, 2024, 12:55 PM

#

In obos::arch::InitializePageTables() it initializes the page tables, starting with mapping the HHDM, then it remaps the kernel based on permissions set in the PHDRs of the kernel's elf image

#

The problem is that limine's magical request sections are getting zeroed

#

I did circumvent this problem before by using limine's physical pages for the kernel

#

But it seems to fail now

ashen goblet Mar 14, 2024, 12:58 PM

#

What magical request sections

flint idol Mar 14, 2024, 12:59 PM

#

The magical request variables

#

like limine_framebuffer_request

#

That gets cleared for some reason

ashen goblet Mar 14, 2024, 12:59 PM

#

Upon loading your own pts?

flint idol Mar 14, 2024, 12:59 PM

#

Yes

flint idol Mar 14, 2024, 1:00 PM

#

flint idol That gets cleared for some reason

well not just that but others

ashen goblet Mar 14, 2024, 1:00 PM

#

Are you sure you’re mapping them correctly

flint idol Mar 14, 2024, 1:00 PM

#

Well up to now this hasn't failed me

#

So I'd assume so

ashen goblet Mar 14, 2024, 1:01 PM

#

What did you change that made it break

flint idol Mar 14, 2024, 1:01 PM

#

Compiled and it stopped working

#

kekw

ashen goblet Mar 14, 2024, 1:02 PM

#

Average osdev experience

#

halfmemeright

#

~~Fix it ❌~~ Use a different bootloader ✅

real pecan Mar 14, 2024, 1:03 PM

#

flint idol yeah I'm switching back to hyper, the bootloader I initially used for the rewrit...

finally embracing the right path LULW

flint idol Mar 14, 2024, 1:03 PM

#

If only I had listened to @real pecan when he told me why I shouldn't switch

#

Then I wouldn't be doing this now

real pecan Mar 14, 2024, 1:04 PM

#

lol

flint idol Mar 14, 2024, 1:04 PM

#

I mean I can just git checkout to the last commit with the appropriate config files for Hyper

#

so it's not that hard

real pecan Mar 14, 2024, 1:05 PM

#

ye

ashen goblet Mar 14, 2024, 1:05 PM

#

Never heard of hyper can you link the repo?

flint idol Mar 14, 2024, 1:05 PM

#

https://github.com/UltraOS/Hyper

GitHub

GitHub - UltraOS/Hyper: A modern x86/ARM bootloader supporting both...

A modern x86/ARM bootloader supporting both BIOS and UEFI - UltraOS/Hyper

#

Funny how after a month, I forgot to say limine instead of hyper here:

if (NOT EXISTS ${CMAKE_SOURCE_DIR}/config/limine.cfg)
    message(FATAL_ERROR "No hyper configuration file detected!")
endif()```

real pecan Mar 14, 2024, 1:12 PM

#

lol

sharp pike Mar 14, 2024, 1:18 PM

#

wow ok

#

that's a massive skill issue honestly

#

fix your page tables lol

flint idol Mar 14, 2024, 1:19 PM

#

It isn't my page tables problem?

#

I literally compiled once and it stopped working.

sharp pike Mar 14, 2024, 1:19 PM

#

or whatever, it's definitely your problem

flint idol Mar 14, 2024, 1:20 PM

#

this is what's going on, it uses the physical pages allocated by limine for the kernel and yet the request is getting zeroed:

static void MapKernel(PageMap* pm)
{
    uint8_t* kfile = (uint8_t*)kernel_file.response->kernel_file->address;
    [[maybe_unused]] size_t kfileSize = kernel_file.response->kernel_file->size;
    elf::Elf64_Ehdr* ehdr = (elf::Elf64_Ehdr*)kfile;
    elf::Elf64_Phdr* firstPhdr = (elf::Elf64_Phdr*)(kfile + ehdr->e_phoff);
    uintptr_t kernelBase = 0, kernelTop = 0;
    for (size_t i = 0; i < ehdr->e_phnum; i++)
    {
        if (firstPhdr[i].p_type != elf::PT_LOAD)
            continue;
        if (firstPhdr[i].p_vaddr < kernelBase || !kernelBase)
            kernelBase = firstPhdr[i].p_vaddr;
    }
    kernelTop = kernelBase;
    for (size_t i = 0; i < ehdr->e_phnum; i++)
    {
        if (firstPhdr[i].p_type != elf::PT_LOAD)
            continue;
        uintptr_t prot = vmm::PROT_NO_DEMAND_PAGE;
        if (firstPhdr[i].p_flags & elf::PF_X)
            prot |= vmm::PROT_EXECUTE;
        if (!(firstPhdr[i].p_flags & elf::PF_W))
            prot |= vmm::PROT_READ_ONLY;
        uint32_t nPages = firstPhdr[i].p_memsz >> 12;
        if ((firstPhdr[i].p_memsz % 4096) != 0)
            nPages++;
        uintptr_t physPages = kernel_addr.response->physical_base + (kernelTop - kernelBase);
        uintptr_t base = firstPhdr[i].p_vaddr & ~0xfff;
        kernelTop = (firstPhdr[i].p_vaddr + firstPhdr[i].p_memsz + 0xfff) & ~0xfff;
        for (uintptr_t kBase = base, j = 0; kBase <= kernelTop; kBase += 4096, j++)
            map_page_to(pm, kBase, physPages + j * 4096, prot);
    }
    g_kernelBase = kernelBase;
    g_kernelTop = kernelTop;
}```

sharp pike Mar 14, 2024, 1:22 PM

#

i can barely read this code

flint idol Mar 14, 2024, 1:23 PM

#

this is the code that matters:

uintptr_t physPages = kernel_addr.response->physical_base + (kernelTop - kernelBase);
uintptr_t base = firstPhdr[i].p_vaddr & ~0xfff;
kernelTop = (firstPhdr[i].p_vaddr + firstPhdr[i].p_memsz + 0xfff) & ~0xfff;
for (uintptr_t kBase = base, j = 0; kBase <= kernelTop; kBase += 4096, j++)
    map_page_to(pm, kBase, physPages + j * 4096, prot);```
Likely the culprit of the bug

#

Because the kernel has no way of knowing whether those physical pages actually correspond to that phdr

lean glen Mar 14, 2024, 1:23 PM

#

flint idol It isn't my page tables problem?

literally is

#

works for everybody here

#

and yeah that code isnt very clean

flint idol Mar 14, 2024, 1:24 PM

#

by "clean" do you mean looks good or likely has bugs

#

or both

lean glen Mar 14, 2024, 1:25 PM

#

both

#

it is harder to find bugs when your code sucks

flint idol Mar 14, 2024, 1:27 PM

#

Unless I'm not supposed to align the virtual address in the phdr down to the nearest page, then the code has no known bugs

sharp pike Mar 14, 2024, 1:41 PM

#

have you tried changing those ~0xfff to ~(uintptr_t)0xfff

flint idol Mar 14, 2024, 1:42 PM

#

No

sharp pike Mar 14, 2024, 1:45 PM

#

in any case i can't really debug your code for you

#

all that i can tell you is that it is 99.9999% your fault, not Limine's

#

if you think that moving to Hyper is somehow going to fix this, i don't think it will

#

and even if it does, it's not a fix

#

it's just avoiding the issue altogether and you'll eventually find another bug at some point with something else like this

flint idol Mar 14, 2024, 1:51 PM

#

Well the issue is probably that I'm getting the offset of the physical page wrong in that code while mapping, so the request(s) that reside in the real virtual offset are garbage values

#

Assuming that starting at 0xffffffff80000000 the physical address of pages is the same as the base passed to limine, I'll start debugging to see if I mess up anywhere

#

Whoopsie

real pecan Mar 14, 2024, 2:05 PM

#

did you mess up

flint idol Mar 14, 2024, 2:05 PM

#

Yes

real pecan Mar 14, 2024, 2:05 PM

#

lol

sharp pike Mar 14, 2024, 2:41 PM

#

at this point Limine has been around for long enough and has been used by enough people that most bugs have been ironed out

#

if you find something you think is a Limine bug, the chance is very high it's a you problem

real pecan Mar 14, 2024, 2:52 PM

#

sharp pike if you find something you think is a Limine bug, the chance is very high it's a ...

this is kinda why i usually don't actively advocate hyper LULW
I see how much beginner spam limine is getting where people dont understand how anything works

#

not saying oberrow is one of them but its just a general trend

sharp pike Mar 14, 2024, 3:03 PM

#

often i have to go out of my way to fix stuff for people just to prove a point

real pecan Mar 14, 2024, 3:04 PM

#

this is kinda what I fear having to do lol

sharp pike Mar 14, 2024, 3:04 PM

#

it's a fine line between me trying to prove a point and people just flat out taking advantage lol

real pecan Mar 14, 2024, 3:04 PM

#

yeah

sharp pike Mar 14, 2024, 3:04 PM

#

example https://github.com/limine-bootloader/limine/issues/337

ashen goblet Mar 14, 2024, 3:11 PM

#

damn

errant basin Mar 14, 2024, 8:00 PM

#

sharp pike example <https://github.com/limine-bootloader/limine/issues/337>

😭

real pecan Mar 15, 2024, 10:27 AM

#

@flint idol is it working now

flint idol Mar 15, 2024, 10:29 AM

#

No KEKW

real pecan Mar 15, 2024, 10:30 AM

#

zamn

flint idol Mar 15, 2024, 11:56 AM

#

This code rather skips a physical page while mapping or it accidentally makes two link to the same page

#

This is quite annoying

#

Solution: Rewrite the code for mapping the kernel

#

No way

#

could it have been because of me accidentally using <= instead <

#

IT WAS

#

🤦

#

Anyway, I was making a stack trace with symbol names before this

ashen goblet Mar 15, 2024, 1:06 PM

#

flint idol Anyway, I was making a stack trace with symbol names before this

How did you implement yours

#

Load the symbol table with limines file feature?

flint idol Mar 15, 2024, 1:07 PM

#

Reading the kernel ELF

ashen goblet Mar 15, 2024, 1:07 PM

#

Ok thats even better

#

I thought about making it like this too

#

But I decided against using file request feature

#

So I have a python script to generate code to put the symtab into the executable itself

#

I could also forbid issuing stack traces before having disk drivers initialized and only then enabling them but that kinda defeats the purpose of early stage debugging

flint idol Mar 15, 2024, 1:10 PM

#

I use limine's kernel file request

ashen goblet Mar 15, 2024, 1:10 PM

#

yea ik

flint idol Mar 15, 2024, 1:11 PM

#

imo on panic, you shouldn't be using any drivers, because those have a chance to crash

#

and if a driver crashes, it panics

#

so you get stuck in an infinite loop

ashen goblet Mar 15, 2024, 1:12 PM

#

Nah I meant read the elf via the driver and parse the ymbol table into some dynamic structure

flint idol Mar 15, 2024, 1:12 PM

#

and usually if you have a stack trace with function names, it's a panic condition

flint idol Mar 15, 2024, 1:12 PM

#

ashen goblet Nah I meant read the elf via the driver and parse the ymbol table into some dyna...

oh ok then

flint idol Mar 15, 2024, 2:50 PM

#

I got the new stack trace done:

Stack trace:
        0xffffffff80008f0d: _ZN4obos6logger13panicVariadicEPvPKcP13__va_list_tag+176
        0xffffffff80008e5d: _ZN4obos6logger13panicVariadicEPvPKcP13__va_list_tag+0
        0xffffffff800017bb: _ZN4obos16pageFaultHandlerEPNS_15interrupt_frameE+1385
        0xffffffff8001055b: External Code
        0xffffffff8000508a: _ZN4obos4arch9ProcStartEPNS_9scheduler9cpu_localE+139```

#

I also got starting APs done

real pecan Mar 15, 2024, 2:53 PM

#

Nice

flint idol Mar 15, 2024, 2:54 PM

#

Also visual studio was being a bitch so now I'm using notepad++

#

and the gdb command line

flint idol Mar 15, 2024, 3:27 PM

#

flint idol I also got starting APs done

Time to start coding the scheduler

#

I'll start with making a thread object

#

That stores info about the thread, like the context on last preempt, the thread id, and some other things required for the scheduler

real pecan Mar 15, 2024, 10:13 PM

#

flint idol Time to start coding the scheduler

Nice

#

You're getting closer to the previous rewrite

flint idol Mar 15, 2024, 10:14 PM

#

Ye

flint idol Mar 17, 2024, 7:58 PM

#

I've tested the kernel at it's current state on two test subjects

#

After a bug fix it works on both

#

Before I can get to the scheduler part of the scheduler, I'm implementing an architecture-specific way of register IRQs in the kernel

#

Then I can start work on ~~the scheduler~~ the timer interrupt so premption can happen

#

Then I can work on the scheduler

#

For now I don't know if I want to implement something like that of the previous kernel's scheduler, every core, when the scheduler is called, looks through the priority lists and schedules a thread, or whether I should use core-specific thread queues

real pecan Mar 19, 2024, 6:39 PM

#

@flint idol hows it going

flint idol Mar 19, 2024, 7:21 PM

#

Uhh it just is

#

I mean I've only written the interface for IRQs

#

I'll probably start coding it today

flint idol Mar 22, 2024, 9:15 PM

#

I've got the code for "allocating" an IRQ done

#

I just need to get the IRQ handler done

#

It will just do some stuff like choose what ISR is to be run

#

It does this to allow for multiple devices to use the same interrupt vector

real pecan Mar 22, 2024, 9:21 PM

#

cool

#

keep it up

flint idol Mar 22, 2024, 9:44 PM

#

I got the basic irq handler done

#

Just it can't do deferred work scheduling yet

#

I need to make the scheduler to do this

flint idol Mar 22, 2024, 10:05 PM

#

If I just make it so deferred work scheduling is processed differently from normal threads, so that they are in a separate list in the cpu local structure, I can do dpcs now

#

Wait

#

Of course I need to make an arch-specific function to setup the thread context

#

Too much work

#

I can just use the existing function

#

For one parameter

ornate ginkgo Mar 22, 2024, 10:40 PM

#

flint idol If I just make it so deferred work scheduling is processed differently from norm...

This is how I do it, i treat dpcs/apcs as part of the run level system rather than the scheduler. Makes more sense imo.

flint idol Mar 22, 2024, 10:45 PM

#

I got DPCs implemented

#

Or well setting up DPCs

#

in the IRQ handler

#

I got to test the interface before starting the scheduler

ornate ginkgo Mar 22, 2024, 10:52 PM

#

nice

flint idol Mar 23, 2024, 12:53 AM

#

I'm taking a break over the weekend

#

I'll be back on monday

#

or maybe sunday

#

Probably not saturday

flint idol Mar 23, 2024, 1:40 AM

#

The abstract IRQ interface works.

#

Which is cool because on sunday or monday I can start writing the scheduler

#

I just tested it with the LAPIC timer and it works

#

So I'll commit and push this code

#

8100 loc so far

#

By the time the scheduler is done it should be around 9000

#

I tested on real hardware and it works

#

Not that it shouldn't, it's nothing fancy

ornate ginkgo Mar 23, 2024, 2:53 AM

#

Always nice to see something on real hardware though

flint idol Mar 24, 2024, 8:20 PM

#

I made a function for calibrating the LAPIC timer using the HPET

#

The next step is abstracting that a bit so that it can be used in a platform-independant way

#

Then, I can work on the scheduler

lean glen Mar 24, 2024, 8:40 PM

#

fairly certain both the lapic and hpet are inherently x86-based so not really platform independent

flint idol Mar 24, 2024, 8:41 PM

#

I meant architecture-independent way of starting a timer

ashen goblet Mar 24, 2024, 9:53 PM

#

When next git push?

#

Wanna look at what you did

#

Sounds really really interesting

flint idol Mar 24, 2024, 9:54 PM

#

For the IRQ interface?

#

It's in scheduler-work

ashen goblet Mar 24, 2024, 9:54 PM

#

Oh mb, didnt look at the other branches

#

I didnt branch predict them

#

Sorry for the bad joke

flint idol Mar 27, 2024, 1:19 PM

#

I expect to finish the scheduler and stuff like that in a couple days

#

I made a way to send IPIs of different types

limber wave Mar 27, 2024, 1:45 PM

#

that would be useful if trying to make an assert macro

#

yeah nevermind me responding to such an old message

#

I do not use Discord and am not familiar with it

flint idol Mar 27, 2024, 1:54 PM

#

limber wave that would be useful if trying to make an assert macro

I used them in OBOS_ASSERT

ashen goblet Mar 27, 2024, 1:57 PM

#

flint idol I made a way to send IPIs of different types

do you have some shared structure of kind? or just different vectors

#

I thought about making an ipi queue

#

so urgent ipis are handled instantaneously

#

and less important ones get queued and get processed by per cpu worker threads

flint idol Mar 27, 2024, 2:01 PM

#

ashen goblet I thought about making an ipi queue

That it is in fact what I did

#

With a shared structure

#

It was something like:

struct ipi
{
  enum
  {
    IPI_INVALID = -1,
    IPI_TIMER,
    IPI_SHOOTDOWN,
  } type;
  union
  {
    base_ipi* base;
    timer_ipi* timer;
    shootdown_ipi* shootdown;
  };
  ipi *next, *prev;
};```

#

With *_ipi being defined accordingly to what they do

ashen goblet Mar 27, 2024, 2:07 PM

#

thats cool yeah

#

do you process ipis as soon as they arrive

flint idol Mar 27, 2024, 2:07 PM

#

Depends on the current IRQL

#

FIFO

#

And the first in is the one processed

#

So, it needs to wait for all ipis added before it to be processed

ashen goblet Mar 27, 2024, 2:08 PM

#

how do you know when the last ipi has arrived

#

do you keep a counter

flint idol Mar 27, 2024, 2:12 PM

#

For example:

Someone wants to do a tlb shootdown on a processor.
They set up the IPI structure and queue it in the processor's IPI queue
The LAPIC then sends an IPI. IPI irqs are processed when IRQL < 0x2
Then, the client waits for the IPI to be processed.
When the target cpu finally receives the interrupt, it processes the first IPI in the list, until there are no more IPI interrupts, and therefore no more IPIs in the queue
When the IPI is processed, it's removed from the queue and the 'processed' variable in the struct is set to true

#

I'll push the code soon, I just want to test it and fix some bugs before I do

ashen goblet Mar 27, 2024, 2:14 PM

#

do you block the issuing core while waiting for that processed bit

flint idol Mar 27, 2024, 2:14 PM

#

Yes

#

'Then, the client waits for the IPI to be processed.'

#

It doesn't need to, but to prevent race conditions in the one place where I do do this (for now), it waits

ashen goblet Mar 27, 2024, 2:15 PM

#

oh yeah I misinterpreted that because I tried to tanslate in my head

flint idol Mar 27, 2024, 2:15 PM

#

the target CPU isn't in charge of freeing any structures in the IPI struct

ashen goblet Mar 27, 2024, 2:15 PM

#

i really like that system yeah

flint idol Mar 27, 2024, 9:04 PM

#

I just found out a possible design issue with the IRQ interface. If an IRQ arrives and it goes through the interface, and no one says it's theirs, there is a possibility that it hangs the system if a false negative was reported somewhere

flint idol Mar 27, 2024, 9:24 PM

#

I said that because the IRQ dispatcher didn't send eois, the handler had to, so if there was no handler called, no eoi is sent

#

The solution is to make it send EOIs

#

The IPI interface indeed works

#

I will now push the code

#

Ok done

#

Currently the IPI interface is x86_64-specific

#

I might want to change that

#

https://github.com/oberrow/obos/commit/9961925a1fe26b3eecde78ea6b98da2287535774

GitHub

Add a IPI interface. Make a function to get the LAPIC timer count b...

…ed of a given frequency. Add a way to start the LAPIC timer. Fix a bug with the IRQ interface.

#

So I can finally get to the actual scheduler

real pecan Mar 27, 2024, 9:33 PM

#

Amazing

flint idol Mar 27, 2024, 9:34 PM

#

The scheduler will probably follow a similar algorithm as the previous scheduler, as I don't feel like bothering with CPU-local thread queues

#

And if I ever do want to, it's an implementation detail so it wouldn't require much effort

real pecan Mar 27, 2024, 9:35 PM

#

Yes

flint idol Mar 27, 2024, 9:36 PM

#

flint idol And if I ever do want to, it's an implementation detail so it wouldn't require m...

well not much effort other than actually implementing it in the scheduler

#

The abstract IRQ interface is around 1000 lines, I don't know why

ashen goblet Mar 27, 2024, 9:49 PM

#

you probably felt more comfortable with the whole idea of interrupts this time

#

so you focused on feature richness

#

does that sound reasonable?

flint idol Mar 27, 2024, 9:50 PM

#

Yeah

#

I'll probably check to see the actual difference

#

Headers take quite a lot of space tbh

#

Weirdly, it's only ~360 lines

ashen goblet Mar 27, 2024, 10:51 PM

#

took a few minutes to look through your design, I really like it

flint idol Mar 27, 2024, 10:51 PM

#

ty

flint idol Mar 28, 2024, 12:32 AM

#

I was thinking about how I'm going to do device discovery for the new kernel, and I've decided to only enumerate devices/buses on ACPI. Then the ACPI device code will load any drivers for buses such as the PCI bus, which will then load other drivers for PCI

#

Instead of assuming some buses are there

#

Even though most modern enough computers are pretty much guaranteed to have a PCI bus, I might as well do it that way as it cleans up the code in the main device discovery file

#

Also who knows, the PCI root bus might start at something other than zero, and the only way to know is through the ACPI tables

#

Specifically MCFG

#

Also I could have seperate drivers for PCI and PCIe

#

It allows for many opportunities that would be pretty messy to do all in one or two files

#

and allows me to flex my module loading (when I get to that point)

#

Anyway I'll do this right after the scheduler

#

It'll go like this:

1. Integrate uACPI.
2. Make module loader/interface.
3. Make device and bus discovery code for ACPI.
4. Make PCI (PCIe?) drivers.```

#

Then I'm making the VFS pain

devout niche Mar 28, 2024, 12:39 AM

#

flint idol I was thinking about how I'm going to do device discovery for the new kernel, an...

that's the spirit, i do the same

#

ACPI provides you a beautiful device tree and i hate to see it go to waste

flint idol Mar 28, 2024, 12:40 AM

#

All because there was no good ACPI interpreter

#

until now

#

#1217009725711847465

#

Or well a couple days ago when @real pecan finished the event handling code

devout niche Mar 28, 2024, 12:41 AM

#

i was doing it with LAI but uACPI will be a welcome step up

flint idol Mar 28, 2024, 12:41 AM

#

But before that uACPI was able to parse the device tree, obviously

#

You could probably see a lot of history of uACPI in my old kernel, as when I implemented uACPI it still didn't have much outside interfaces

#

Not even sleep/reboot

#

I had to make some makeshift code which I decided to comment out instead of delete when the helpers did come out, to keep as a souvenir

#

The makeshift ACPI code that's still used in the old kernel is device tree code

#

It's not the cleanest, but it was better than nothing

flint idol Mar 28, 2024, 12:44 AM

#

flint idol It'll go like this: ``` 1. Integrate uACPI. 2. Make module loader/interface. 3. ...

I think this is a bit out of order, as I might need the VFS to enumerate drivers that can be loaded

#

But I can still make the code as long as I provide it a list of driver binaries

#

After the VFS, I can hopefully start making user mode stuff

#

So in a month or two

#

or three

inland radish Mar 28, 2024, 5:44 AM

#

flint idol The scheduler will probably follow a similar algorithm as the previous scheduler...

CPU local queues are easy to do, why not

#

It's just re-enqueue the thread on the current CPU's queue instead of the global queue

#

If you want load balancing you can make other cores try to steal from each other if they don't have threads to run at the priority of the scheduled out thread or higher

flint idol Mar 28, 2024, 10:43 PM

#

I've written the scheduler

#

I just need to test it

flint idol Mar 28, 2024, 11:08 PM

#

#

Anyway I found a bug with the slab allocator

#

idk what it is, but it's causing a page fault

#

ok what is this

#

The fault is happening as a result of a nullptr deref at line 353 which is this:

while (currentNode->next != stepNode)

but right before that, it checks for nullptr

#

if (!currentNode)
    break;```

vernal chasm Mar 29, 2024, 1:34 AM

#

flint idol The fault is happening as a result of a nullptr deref at line 353 which is this:...

It's spooky actions at a distance O_O

flint idol Mar 29, 2024, 2:29 PM

#

vernal chasm It's spooky actions at a distance O_O

Is that supposed to mean that something isn't locked?

#

In the slab allocator?

#

Hmmm

#

Well the function for allocating from a region node does lock it when it needs to sort

#

So that's not the problem

#

Maybe it's just that I misread the address

#

I did not

#

I should fix my stack trace first, it page faults for some unknown reason

flint idol Mar 29, 2024, 3:03 PM

#

Stack trace fixed

#

I also fixed a bug in the VMM

#

where the HHDM wasn't registered as used pages in the kernel context

#

Even though it should've been

#

I just fixed the bug in the allocator

#

I was reading the code backwards because loops are goofy it seems

#

The condition checking of the loop was after a call to swapNodes, but I hadn't noticed that in the first place, so I thought it was after an iteration of the loop

#

There seems to be a bug with IPIs when it's being sent to another processor

#

idk what it is

#

It was a missing sti in the processor's entry point

#

#

Uhh

#

What?

real pecan Mar 29, 2024, 3:37 PM

#

That stackttace tho

flint idol Mar 29, 2024, 3:38 PM

#

It was in the YieldThread() function which doesn't save a frame pointer

#

So the stack trace was poopy

#

Just YieldThread() shouldn't have been called

#

with a nullptr argument

#

if (!GetCPUPtr()->currentThread)
    schedule();
arch::YieldThread(GetCPUPtr()->currentThread);```

#

The only place that calls YieldThread()

#

in yield()

#

For some dumb reason, panics that aren't on the BSP are painfully slow

#

It's probably some problem with caching

#

I made it panic if schedule() returns, and it did

#

I should make it panic unless g_initialized is false, which is the only case where that's possible

flint idol Mar 29, 2024, 5:33 PM

#

What the fuck is happening? These two lines somehow cause a cpu exception somewhere else in the code:

if (oldIRQL != 0xff)
    LowerIRQL(oldIRQL);```
Except that part of the kernel is **completely** unrelated to this part and the scheduler

#

and on top of that, it's somehow making the panic function shit itself

#

This better not be a bug with the VMM

#

Commenting them out gets rid of the crash

#

Uncommenting them gives the crash once again

#

How can I make code so utterly fucked, that two completely unrelated parts of the kernel can cause the other part to crash

#

For reference, this was in the timer IRQ handler for the scheduler

#

The part that crashed was somewhere in the VMM initialization code

#

It's crashing while logging a string

#

Which isn't even the correct string?

#

"%s: Initializing page node allocator.\n" was passed

#

"e page descriptors for page %p.\n" was the parameter it got

#

It's a bug with mapping the kernel

#

https://tenor.com/view/sad-no-disappear-sad-noob-gif-24308846

Tenor

#

Conclusion: This bug should be impossible, but it happened anyway

#

I think I found an oversight/bug/whatever with limine mapping the kernel. I don't know how to explain it, so here is a snippet of an info tlb command:

ffffffff80011000: 000000001feaa000 ---------
ffffffff80012000: 000000001feab000 ----A----
ffffffff80014000: 000000001fead000 X---A----
ffffffff80015000: 000000001feae000 X---A----```
For some reason, the linker, while linking my kernel, skipped a page between `0xffffffff80012000` and `0xffffffff80013000`. Limine saw this, so it obviously it didn't map the page, but it skipped the physical page that would've been at `0xffffffff80013000` if it were mapped

#

Idk why this happens, but it'll be an easy fix in my kernel

#

if ((base - kernelTop) >= 0x1000)
    physPages += (base - kernelTop);```
In the kernel mapping code fixed it

#

Maybe I should just retrieve the physical page for each virtual address to be mapped and map that instead of playing catch-up with limine's map function doing weird side-cases

#

Now I don't have to worry about goofy things happening because of the physical page not being correct

#

So, back to debugging the timer interrupt only ever arriving once

#

Btw, I make sure an EOI is sent for all cases

#

and the IRQL is zero, which means all interrupts are unmasked

#

IF in EFLAGS is set

#

The initial count is in fact set

#

EOI is sent in the timer IRQ handler before the scheduler is called

#

If for some reason, the IRQ handler never gets called by the IRQ dispatcher, an EOI is sent by the IRQ dispatcher

#

ISR and IRR are 0x20, the timer IRQ

#

PPR is 0x20

#

The only reason the ISR (interrupt service register) in the LAPIC would be 0x20 when the interrupt was already serviced is if an EOI wasn't sent

#

Maybe some weird caching thing is happening and a write to the EOI register is doing nothing

#

Even though I marked the LAPIC registers as UC

#

According to info tlb, the LAPIC registers are UC

#

Conclusion: The LAPIC doesn't like me

flint idol Mar 29, 2024, 6:27 PM

#

flint idol PPR is 0x20

TPR is 0x00

#

No calls to SendEOI are optimized out

#

Uhhh

#

What's happening?

#

Looking at ESR, bit 7 is set

#

Meaning "Illegal Register Address"

real pecan Mar 29, 2024, 6:35 PM

#

Bruh

flint idol Mar 29, 2024, 6:36 PM

#

I just love it when all conditions for a timer interrupt to happen are met, but no timer interrupt happens

#

I used GDB to send an EOI, and an interrupt happens

#

Except it only happens once

#

I don't know why the hell my code isn't sending an EOI, even though arch::SendEOI(f); unconditionally sends an EOI

#

Of course when I go into a debugger to debug the EOI not being sent, the EOI is sent KEKW

#

I made sure that an EOI is sent at the beginning of the IRQ dispatcher, and the bug still persists

dire belfry Mar 29, 2024, 6:59 PM

#

didn't expect people to be typing in here when i looked

flint idol Mar 29, 2024, 7:00 PM

#

With people being me

#

Typing my debugging so I don't forget anything

flint idol Mar 29, 2024, 10:27 PM

#

The scheduler works

#

Which is good

flint idol Mar 29, 2024, 10:28 PM

#

flint idol The scheduler works

on one core at least

#

Four cores make qemu shit itself

#

I think that was because of the trace options

#

Now it fails raising the IRQL

#

That's weird.

#

uint8_t oldIRQL = 0xff;
if (GetIRQL() < 2)
    RaiseIRQL(2, &oldIRQL, false);```
causes:
`Attempt to call RaiseIRQL() with the irql 2, which is less than the current IRQL, 15.`

real pecan Mar 29, 2024, 10:33 PM

#

Add more log statements

real pecan Mar 29, 2024, 10:34 PM

#

flint idol ```c++ uint8_t oldIRQL = 0xff; if (GetIRQL() < 2) RaiseIRQL(2, &oldIRQL, fal...

Is it possible for irql to change after the jf statement runs?

#

If this code gets preempted

flint idol Mar 29, 2024, 10:35 PM

#

Probably if I forget to lower the IRQL in an interrupt handler and it returns (likely)

real pecan Mar 29, 2024, 10:36 PM

#

Yup

flint idol Mar 29, 2024, 10:37 PM

#

I think I'm accidentally having a repeat of the 2nd rewrite of the kernel. An unstable scheduler.

#

The bug only happens sometimes

#

I better fix that instead of ignoring it

#

The scheduler itself is stable, just the context saving/switching functions seem to be unstable for some reason

real pecan Mar 29, 2024, 10:40 PM

#

You should

#OBOS (not vibecoded)