OBOS (not vibecoded) | Operating System Development | Page 4

flint idol Mar 29, 2024, 10:54 PM

#

Fixing the timer calibrate function fixed the scheduler

#

I think the timer was running way too fast

#

Still got some bugs to fix

#

But then I can move onto the driver interface

#

Which consequencently means I implement uACPI

ornate ginkgo Mar 29, 2024, 10:59 PM

#

Yeah lol, average c++ stack trace: wider than it is deep

real pecan Mar 29, 2024, 11:00 PM

#

flint idol I think the timer was running *way* too fast

Yeah that still shouldn't cause panics tho

flint idol Mar 29, 2024, 11:05 PM

#

ornate ginkgo Yeah lol, average c++ stack trace: wider than it is deep

Lol

flint idol Mar 29, 2024, 11:05 PM

#

real pecan Yeah that still shouldn't cause panics tho

It causes everything to shit itself, including the host os KEKW

vernal chasm Mar 29, 2024, 11:10 PM

#

ornate ginkgo Yeah lol, average c++ stack trace: wider than it is deep

meanwhile in c after breaking something you just get the same stack frame duped a few million times xD

flint idol Mar 29, 2024, 11:25 PM

#

flint idol Still got some bugs to fix

Which of course have to do with stack corruption

flint idol Mar 30, 2024, 1:48 PM

#

The stack corruption was caused by a missing cli in the function to switch to another context

#

I added a watchdog timer after initializing the scheduler to make sure the LAPIC timer doesn't hang

vernal chasm Mar 30, 2024, 1:56 PM

#

flint idol The stack corruption was caused by a missing `cli` in the function to switch to ...

Oof

flint idol Mar 30, 2024, 2:00 PM

#

Well there is a bug with the other CPUs hanging in the cpu entry point because their IRQL is 0xf

#

But it only happens to some CPUs

ashen goblet Mar 30, 2024, 2:35 PM

#

The whole irql concept sounds a bit bug prone

#

But it‘s a nice flex once you got it right

flint idol Mar 30, 2024, 2:35 PM

#

ashen goblet The whole irql concept sounds a bit bug prone

It indeed is

#

I'll get it right eventually

#

I suspect the bug is with the IPI handler not lowering the IRQL

#

Somehow

#

It's not the IPI handler

#

So it's likely the scheduler

#

All I know is that an asynchronus context is raising the IRQL to 0xf and not lowering it

flint idol Mar 30, 2024, 3:20 PM

#

The bug was a missing LowerIRQL and the end of the irq dispatcher

#

It did lower the IRQL when calling the handler

#

But only then

#

So, if nothing recognized that interrupt, it would leave the IRQL at 0xf

#

I'll do one last test on the scheduler, then I'll commit and merge the branch

#

Why the fuck is there a stack trace with 790 frames

#

For future reference, do not put assertions in *IRQL functions

#

At least not this assertion:

OBOS_ASSERTP(arch::GetIRQL() == getIRQLVar(), "");

#

Now there is a bug with the IRQL not getting lowered from 0x2

#

Well I said that after implementing a fix

#

But the scheduler seems to be stable without any apparent bugs

real pecan Mar 30, 2024, 3:32 PM

#

flint idol At least not this assertion: ```c++ OBOS_ASSERTP(arch::GetIRQL() == getIRQLVar()...

Why

real pecan Mar 30, 2024, 3:33 PM

#

flint idol But the scheduler seems to be stable without any apparent bugs

Now try at max frequency

flint idol Mar 30, 2024, 3:33 PM

#

Well, locks raise the IRQL, right?

#

The panic function calls vprintf which locks the printf lock

#

which therefore raises the irql

#

but, that assertion fails again

#

and that happens infinitely until the stack guard page is hit

flint idol Mar 30, 2024, 3:34 PM

#

real pecan Now try at max frequency

Well, what is max frequency?

#

The previous bug with the timer being fast was because of a bug with the calibration function

#

I'll try at 1 khz

#

Shits itself

#

more like: it found a bug

#

Attempt to call LowerIRQL() with the irql 2, which is greater than the current IRQL, 15.
Stack trace:
        0xffffffff8000acbb: _ZN4obos6logger13panicVariadicEPvPKcP13__va_list_tag+189
        0xffffffff8000abfe: _ZN4obos6logger13panicVariadicEPvPKcP13__va_list_tag+0
        0xffffffff8000adc0: _ZN4obos9LowerIRQLEhb+131
        0xffffffff800016d0: _ZN4obos16pageFaultHandlerEPNS_15interrupt_frameE+343
        0xffffffff80013b20: External Code
        0x0000000000000003: External Code```

#

What

#

That makes no sense

#

if (newIRQL > getIRQLVar())
            logger::panic(nullptr, "Attempt to call %s() with the irql %d, which is greater than the current IRQL, %d.\n", __func__, newIRQL, s_irql);

#

It triple faults because cr3=0 at 0.1 kHz

#

wait

#

I got my numbers backwards

flint idol Mar 30, 2024, 3:41 PM

#

flint idol I'll try at 1 khz

*1 mHz

flint idol Mar 30, 2024, 3:41 PM

#

flint idol It triple faults because cr3=0 at 0.1 kHz

*0.1 mHz

#

I somehow crashed qemu at 10 kHz

#

[Log] KernelArchInit: Initializing scheduler.
**
ERROR:../../../system/cpus.c:504:qemu_mutex_lock_iothread_impl: assertion failed: (!qemu_mutex_iothread_locked())
Bail out! ERROR:../../../system/cpus.c:504:qemu_mutex_lock_iothread_impl: assertion failed: (!qemu_mutex_iothread_locked())```

#

I think keeping it at a couple kHz is a better choice

#

I'll keep it at 4kHz

#

I just need to fix the bug where cr3=nullptr

#

Then I can commit

#

Crashed qemu again KEKW

#

A thread that wasn't fully setup yet was recognized by the scheduler

#

and tried to run before it's context was setup

#

g_threadPriorities[(int)thr->priority].Append(thr);
arch::SetupThreadContext(&thr->context, &thr->thread_stack, (uintptr_t)entry, 0, false, 0x10000, thr->addressSpace);

lean glen Mar 30, 2024, 3:56 PM

#

setup the thread before

flint idol Mar 30, 2024, 3:56 PM

#

Yeah I know

#

Now that there are no apparent bugs with the scheduler/irqs/context switching/whatever

#

I can commit

#

Committed the code

#

So now it's time to implement the driver interface

#

I'll start in a couple hours by porting uACPI

#

I got something to do now

real pecan Mar 30, 2024, 4:17 PM

#

Nice

real pecan Mar 30, 2024, 4:17 PM

#

flint idol Now that there are no apparent bugs with the scheduler/irqs/context switching/wh...

Now it works at any frequency?

flint idol Mar 30, 2024, 4:17 PM

#

No

#

Because qemu crashes at higher frequencies

#

or is really unreliable

real pecan Mar 30, 2024, 4:18 PM

#

Qemu or your scheduler KEKW

flint idol Mar 30, 2024, 4:18 PM

#

Both KEKW

real pecan Mar 30, 2024, 7:54 PM

#

@flint idol FYI i just merged the branch with a bunch of improvements for uacpi

flint idol Mar 30, 2024, 7:55 PM

#

Of course you did...

#

Ok I'll change the git tag for fetch content

real pecan Mar 30, 2024, 7:59 PM

#

dont worry it doesnt change anything public facing

flint idol Mar 30, 2024, 7:59 PM

#

@real pecan What IRQL should ACPI events run at?

#

Should they be pre-emptible?

real pecan Mar 30, 2024, 7:59 PM

#

yeah

#

they can sleep or acquire mutexes or do anything

flint idol Mar 30, 2024, 7:59 PM

#

So unmasked?

real pecan Mar 30, 2024, 7:59 PM

#

normal dispatcher level

flint idol Mar 30, 2024, 8:00 PM

#

Ok

#

What IRQL should the IRQ be?

real pecan Mar 30, 2024, 8:00 PM

#

the normal irq level

flint idol Mar 30, 2024, 8:00 PM

#

Like should a IRQL 2 task be able to get an interrupt from ACPI

#

For example

real pecan Mar 30, 2024, 8:01 PM

#

it doesnt do much besides reading registers and potentially scheduling some events

real pecan Mar 30, 2024, 8:01 PM

#

flint idol Like should a IRQL 2 task be able to get an interrupt from ACPI

whats IRQL 2?

flint idol Mar 30, 2024, 8:01 PM

#

IRQL 2 is used for the scheduler

real pecan Mar 30, 2024, 8:01 PM

#

sure why not

lean glen Mar 30, 2024, 8:01 PM

#

the scheduler is typically a DPC

real pecan Mar 30, 2024, 8:01 PM

#

acpi irq is not any different from any other device

#

also if u want to eventually implement support for EC u can refer to this https://github.com/managarm/managarm/blob/master/kernel/thor/system/acpi/ec.cpp

GitHub

managarm/kernel/thor/system/acpi/ec.cpp at master · managarm/managarm

Pragmatic microkernel-based OS with fully asynchronous I/O - managarm/managarm

#

EC is used for anything like lid close/open events, charger plug/unplug, fn+sleep/brightness/wahtever

#

power button on some hardware as well

#

Oh and this is managarm's implementation of kernel api:
https://github.com/managarm/managarm/blob/master/kernel/thor/system/acpi/glue.cpp

GitHub

managarm/kernel/thor/system/acpi/glue.cpp at master · managarm/mana...

Pragmatic microkernel-based OS with fully asynchronous I/O - managarm/managarm

#

might also be useful

flint idol Mar 30, 2024, 8:13 PM

#

Well I'd just be using my old kernel as a reference for that

real pecan Mar 30, 2024, 8:13 PM

#

sure

flint idol Mar 30, 2024, 8:17 PM

#

@real pecan what is uacpi_cpu_flags used for?

real pecan Mar 30, 2024, 8:17 PM

#

spinlock api

#

u can just typedef it to irql

#

(if thats how ur spinlocks work)

flint idol Mar 30, 2024, 8:19 PM

#

oh ok

flint idol Mar 31, 2024, 4:57 PM

#

I'm currently fighting with cmake and ld to properly link against the static library that has uACPI

flint idol Mar 31, 2024, 5:26 PM

#

It was because I didn't reference uACPI anywhere in the kernel

real pecan Mar 31, 2024, 5:27 PM

#

@flint idol BTW you're not allowed to use memcpy for raw memory r/w

flint idol Mar 31, 2024, 5:27 PM

#

I'm not

#

Oof

#

I'll fix that

real pecan Mar 31, 2024, 5:28 PM

#

I should really specify it in API description

flint idol Mar 31, 2024, 5:28 PM

#

I'd assume because the hardware might expect a write/read at a certain granularity

real pecan Mar 31, 2024, 5:29 PM

#

Yea

flint idol Mar 31, 2024, 8:35 PM

#

@real pecan is the mutex for the kernel api for uACPI recursive or not?

real pecan Mar 31, 2024, 8:36 PM

#

if only you read kernel_api.h caret

flint idol Mar 31, 2024, 8:36 PM

#

oh yeah

#

didn't see that

#

so do you want me to hang or fail if the current thread already has taken the mutex?

#

Actually that's a bad question

real pecan Mar 31, 2024, 8:38 PM

#

i mean that would be a bug in uacpi

#

but idk

#

you could log it at least

real pecan Mar 31, 2024, 8:58 PM

#

@flint idol i fixed the PRIX64 thing

flint idol Mar 31, 2024, 8:58 PM

#

ok

real pecan Mar 31, 2024, 8:58 PM

#

..and an out of bounds found by qookie fuzzer

flint idol Apr 1, 2024, 4:20 PM

#

I finished implementing the kernel API for uACPI

#

Time to test it

real pecan Apr 1, 2024, 4:21 PM

#

Do it

flint idol Apr 1, 2024, 6:25 PM

#

@real pecan

#

uACPI seems to be freeing an object that was rather:

already freed
never allocated
gotten from a different source than uacpi_kernel_alloc

real pecan Apr 1, 2024, 6:29 PM

#

Highly doubt it

#

If this is qemu its blob has been tested a million times

flint idol Apr 1, 2024, 6:30 PM

#

Maybe the allocator expects the exact base address of the allocated address

#

And uACPI provided the base + some offset

real pecan Apr 1, 2024, 6:31 PM

#

UACPI frees by specifying the exact address that was given from malloc

flint idol Apr 1, 2024, 6:31 PM

#

well that's not the case then

real pecan Apr 1, 2024, 6:31 PM

#

Well if it works in managarm then I dont see why it wouldn't work here

#

Test managarm with this exact command line

flint idol Apr 1, 2024, 6:32 PM

#

I'll look into the allocator structures myself and see the allocator somehow missed the object in it's allocated list

real pecan Apr 1, 2024, 6:32 PM

#

Or log every allocation

#

And see if it was freed at some point

#

Or whatever

flint idol Apr 1, 2024, 6:34 PM

#

It seems to be memory corruption in the allocator

#

Doubt it's uACPI

#

I'm probably providing an invalidly sized block

real pecan Apr 1, 2024, 6:36 PM

#

ultrameme

flint idol Apr 1, 2024, 6:37 PM

#

I just don't know why it would fail now out of all times

real pecan Apr 1, 2024, 6:38 PM

#

uACPI really stresses it probably

flint idol Apr 1, 2024, 6:39 PM

#

and tbf this allocator is only less than a month old and I'm stressing it with a whole-ass acpi library

#

The previous kernel's allocator was a couple months old, and stress tested a lot

#

or at least stressed a lot more

real pecan Apr 1, 2024, 7:10 PM

#

A very trivial slab cache for uACPI objects could really help here btw, ill probably make an optional interface for that

flint idol Apr 1, 2024, 7:21 PM

#

So, there's this one line

newNode->data = (char*)ROUND_UP(((uintptr_t)(newNode + 1) - sizeof(uintptr_t)), padding);

for setting the data pointer of a new node
I'm wondering, what is the point of the - sizeof(uintptr_t) after adding the offset to the node?
I remove it, and the entire kernel falls into shambles
The only thing I can think of it doing is making the pointer leak into the node structure, which isn't what happens with that line

#

Well it is rounding up

#

The node does indeed exist in the allocator's structures thonk

#

So, it's these lines that are causing the allocator to skip the region node containing the node:

if (!(base >= cregion->base && base < ((char*)cregion->base + cregion->regionSize)))
    goto bottom;```

#

cregion->regionSize=0x280
cregion->base=0xffff8000fef98000
addr=0xffff8000fef982b0

#

the region size is incorrect

#

the region size is calculated without considering the region node size

real pecan Apr 1, 2024, 7:46 PM

#

Damn

flint idol Apr 1, 2024, 8:18 PM

#

I've fixed the bug in the allocator

#

and now uACPI initializes properly*
*except for PCI stuff, I don't support that yet

real pecan Apr 1, 2024, 8:18 PM

#

Nice

flint idol Apr 1, 2024, 8:19 PM

#

Might as well implement PCI register accesses now

#

Done:

#

Time to test shutdown and stuff

#

Then, I'm moving onto the module loader

#

Shutdown works

#

Reboot works

#

I'm not going to test anything else

#

Anyway time for a lot of copy+paste from the old kernel

#

Will a lot being the relocation code

#

But, before that, I'll push the new code

#

Despite my original plans for this kernel's driver interface (to just copy+paste the old one), I'm going to redo it

real pecan Apr 1, 2024, 8:43 PM

#

Awesome

flint idol Apr 1, 2024, 8:43 PM

#

After this, I'm making the VFS

#

I think this'll take a month or two

real pecan Apr 1, 2024, 8:43 PM

#

Damn

lean glen Apr 1, 2024, 8:44 PM

#

thats a long time

flint idol Apr 1, 2024, 8:44 PM

#

It's not going to be complicated necessarily

#

I probably won't have that much time

#

bcuz school and stuff

flint idol Apr 2, 2024, 11:55 AM

#

I just tested on real hardware, it breaks while initializing the scheduler

ashen goblet Apr 2, 2024, 12:14 PM

#

Unitialized memory maybe?

flint idol Apr 2, 2024, 12:18 PM

#

Idk

#

I'll debug it later

#

On another computer, it pages faults in uACPI because of an invalid pointer from the allocator

#

So there's a bug with the allocator and the scheduler

ashen goblet Apr 2, 2024, 12:38 PM

#

||allocator bugs are the worst||

real pecan Apr 2, 2024, 12:46 PM

#

thats why you write userspace tests for your allocator instead of testing it against an actual kernel in runtime

flint idol Apr 2, 2024, 12:54 PM

#

No

flint idol Apr 2, 2024, 8:42 PM

#

Right so remember how I said I wouldn't test my allocator

#

It would be a lot easier to fix it if I just tested it

#

It'll be something like:

size_t passes = 2097152;
void* lastDiv16Pointer = nullptr;
for (size_t i = 0; i < passes; i++)
{
    void* mem = allocator.Allocate(1 /* The allocator counts in chunks of the block size, not bytes. */);
    if (i % 16)
    {
        if (lastDiv16Pointer)
            allocator.Free(lastDiv16Pointer);
        lastDiv16Pointer = mem;
    }
}```

#

That will stress test the allocator with a lot of allocations, whilst only freeing some of the allocated pointers

#

To simulate the kernel's allocations

#

A lot aren't freed because they needn't be freed

#

or they're for drivers that the kernel needs

#

etc.

#

Actually I might as well use that code

flint idol Apr 2, 2024, 8:48 PM

#

flint idol That will stress test the allocator with a lot of allocations, whilst only freei...

it also allows me to break on the exact allocation that page faults or reports allocator corruption

#

because I can just break when i == (whatever index failed)

#

The allocator isn't randomized in any way

real pecan Apr 2, 2024, 9:21 PM

#

you should really randomize it tho

#

just generate a random seed and print it at the beginning

#

then feed it to the generator

#

then generate allocation sizes

#

they will be the same for the same seed so its reproducable

flint idol Apr 2, 2024, 10:09 PM

#

Why must the allocator not fail the one moment I need it to fail?

#

Maybe it's a problem with non-zeroed memory

#

I'll make limine randomize the memory and then try

real pecan Apr 2, 2024, 10:11 PM

#

U can just dd rand to a file in /tmp

#

And then make qemu use that file as ram

flint idol Apr 2, 2024, 10:12 PM

#

wait I can

#

what argument to qemu

real pecan Apr 2, 2024, 10:12 PM

#

object memfile or something like that idr, Google it

#

https://unix.stackexchange.com/questions/616596/mapping-guest-ram-to-file-in-qemu

Unix & Linux Stack Exchange

Mapping guest RAM to file in qemu

We're emulating a Cortex M3 cpu and would like to pass some parameters to the guest during run-time. The simplest idea seems to be to write directly to some memory area. I tried simply adding -mem-...

#

object memory-backend-file,id=mem,size=128K,mem-path /tmp/qemu.ram \

flint idol Apr 2, 2024, 10:20 PM

#

That didn't work

#

"invalid object type: memory-backend-file"

real pecan Apr 2, 2024, 10:22 PM

#

flint idol "invalid object type: memory-backend-file"

Perhaps its called something else now

#

Should work tho...

#

Show full command line

flint idol Apr 2, 2024, 10:23 PM

#

It doesn't

#

qemu-system-x86_64 ^
-drive file=out/obos.iso,format=raw ^
-m 1G ^
-gdb tcp:0.0.0.0:1234 -S ^
-accel tcg ^
-cpu "Haswell" ^
-M q35 ^
-monitor stdio ^
-debugcon file:CON ^
-serial tcp:0.0.0.0:1534,server,nowait ^
-smp cores=8,threads=1,sockets=1 ^
-d int ^
-D qemu_log.txt ^
-object memory-backend-file,id=mem1,share=on,mem-path=./memory.img,size=1G```

real pecan Apr 2, 2024, 10:23 PM

#

Oh that's Linux only

#

Probably

flint idol Apr 2, 2024, 10:23 PM

#

ok then to wsl I go

real pecan Apr 2, 2024, 10:23 PM

#

Try it

#

And that img should be in /tmp or your perf will be 0

flint idol Apr 2, 2024, 10:25 PM

#

it is indeed linux only

#

and KVM fails with "KVM: entry failed, hardware error 0x0" upon initialization of my scheduler

#

rather right after initialization of the scheduler

real pecan Apr 2, 2024, 10:26 PM

#

Yah something is fucked in your kernel

flint idol Apr 2, 2024, 10:26 PM

#

https://tenor.com/view/always-has-been-among-us-astronaut-space-betrayal-gif-23836476

Tenor

real pecan Apr 2, 2024, 10:26 PM

#

Lol

flint idol Apr 2, 2024, 10:26 PM

#

and... of course debugcon doesn't work

#

nvm

#

the output file was invalid

real pecan Apr 2, 2024, 10:27 PM

#

Yeah only d int doesn't work

flint idol Apr 2, 2024, 10:28 PM

#

instead of fixing the underlying bug in my kernel that caused kvm to break, I used TCG

real pecan Apr 2, 2024, 10:28 PM

#

Lol

#

That's one way to solve it

flint idol Apr 2, 2024, 10:31 PM

#

this is pretty slow

#

it took like a minute for 60000 of passes

#

I'll blame it on qemu tbh

#

qemu on wsl is pretty slow

real pecan Apr 2, 2024, 10:33 PM

#

Is your ram in tmp?

flint idol Apr 2, 2024, 10:33 PM

#

60000 passes on windows qemu took less than 20 seconds

flint idol Apr 2, 2024, 10:33 PM

#

real pecan Is your ram in tmp?

yes

real pecan Apr 2, 2024, 10:33 PM

#

Wait this is TCG?

flint idol Apr 2, 2024, 10:33 PM

#

yes

real pecan Apr 2, 2024, 10:33 PM

#

I mean lmao

#

TCG is more of a toy

#

Also you did spray the ram right?

flint idol Apr 2, 2024, 10:35 PM

#

yes

real pecan Apr 2, 2024, 10:35 PM

#

From dev random?

flint idol Apr 2, 2024, 10:35 PM

#

from /dev/urandom

real pecan Apr 2, 2024, 10:35 PM

#

Thats fine I guess

flint idol Apr 2, 2024, 10:37 PM

#

340000 passes and still no crash?

#

Why mustn't it fail when I need it too?

real pecan Apr 2, 2024, 10:38 PM

#

Is it allocating random amounts?

flint idol Apr 2, 2024, 10:38 PM

#

Well

#

Almost

#

it allocates i % 16 every time

#

and frees an object every 16 iterations

real pecan Apr 2, 2024, 10:39 PM

#

Anyway the way you're stressing it is too static to trigger a bug

flint idol Apr 2, 2024, 10:40 PM

#

The bug I'm trying to trigger is specifically a page fault that happens when the allocator gives out a region outside of a region's bounds

#

I'll just get a rand implementation

real pecan Apr 2, 2024, 10:40 PM

#

randread

flint idol Apr 2, 2024, 10:40 PM

#

rdseed?

real pecan Apr 2, 2024, 10:41 PM

#

Or wait whatever the insn was called

#

Rdrand

real pecan Apr 2, 2024, 10:41 PM

#

flint idol `rdseed`?

That and then rdrand

flint idol Apr 2, 2024, 10:42 PM

#

well, haswell doesn't seem to have that

#

maybe I'm mistaken

#

nvm

#

I misread ecx as eax

real pecan Apr 2, 2024, 10:43 PM

#

It definitely does

#

All 64 bit CPUs do I think

#

Or maybe not all

flint idol Apr 2, 2024, 10:46 PM

#

How convinient KEKW

#

It fails on the second iteration

#

nvm, allocate failed because of the size being zero

real pecan Apr 2, 2024, 10:48 PM

#

Lol

flint idol Apr 2, 2024, 10:51 PM

#

I don't even care about what I was doing before. I'm optimizing this allocator

real pecan Apr 2, 2024, 10:51 PM

#

Do it

flint idol Apr 2, 2024, 10:51 PM

#

Sorting lists on. every. single allocation is painfully slow

#

and bubble sort isn't neccessarily the fastest sorting algorithm

real pecan Apr 2, 2024, 10:52 PM

#

Wtf

flint idol Apr 2, 2024, 10:52 PM

#

Don't ask KEKW

real pecan Apr 2, 2024, 10:52 PM

#

Anyway off to bed but good luck with your debugging

flint idol Apr 2, 2024, 10:53 PM

#

ty

real pecan Apr 3, 2024, 7:10 AM

#

@flint idol why is this returning a constant number?

    uacpi_u64 uacpi_kernel_get_ticks(void)
    {
        return 2500; // 4000 hz -> 250000 ns = 2500 ticks.
    }

vernal chasm Apr 3, 2024, 7:53 AM

#

real pecan <@841762704766140486> why is this returning a constant number? ```cpp uacpi_...

We all love our magic numbers kekw

flint idol Apr 3, 2024, 9:35 AM

#

real pecan <@841762704766140486> why is this returning a constant number? ```cpp uacpi_...

I saw what my previous kernel did and assumed it was right

real pecan Apr 3, 2024, 9:36 AM

#

Lol

flint idol Apr 3, 2024, 9:36 AM

#

I just woke up now so I'll fix it later

flint idol Apr 3, 2024, 1:51 PM

#

I think I should fix make it so my IRQ interface so that multiple handlers can be called for the same IRQ source

#

By default, if an IRQ comes, the first IRQ object to recognize the IRQ is chosen, and only that one

#

Which isn't ideal, especially for the timer interrupt

#

It technically isn't linked to the scheduler, but if it's the only handler that gets run, it basically is

inland radish Apr 3, 2024, 1:56 PM

#

flint idol Which isn't ideal, especially for the timer interrupt

What do you mean?

#

Are you going to have multiple interrupt handlers for one timer?

flint idol Apr 3, 2024, 1:57 PM

#

Kinda

#

It's more like a list for irq handlers for each irq object

#

and then the irq dispatcher would call/dispatch to them all to DPCs

#

I just said the timer IRQ as an example

#

This will allow for timer objects which have a callback

#

for example, because the scheduler won't be the only thing that can run off the timer interrupt

ornate ginkgo Apr 3, 2024, 9:37 PM

#

You might find it useful to abstract away the timer into a queue of events, which the timer drives. Your scheduler and other things can put things in the queue to expire at certain times.

flint idol Apr 3, 2024, 9:37 PM

#

That might be a better way to do what I wanted to

flint idol Apr 4, 2024, 9:38 PM

#

Well I just barely optimized the allocator

#

in terms of speed

#

20000 allocations with randomly sized allocations from 16-256 bytes

#

took a minute

#

Actually if anything, I slowed it down

#

By a lot

#

Why is the version that theoretically should be doing less work a lot slower

#

Ok well one difference is the newer code has one region node that has all the nodes

#

the other has 208 by the end

#

Wait that should be impossible

#

there are 4096 nodes preallocated

#

and by the 20,000th pass, there was only one region node

ashen goblet Apr 4, 2024, 10:17 PM

#

What kind of nodes are you referring to? For slab or smth?

flint idol Apr 4, 2024, 10:17 PM

#

Slab

#

So, for reasons unknown to mankind, the version that should be slower is faster than the one that should be faster

#

When all that was added was some booleans to skip doing work that was already done

#

or didn't need to be done

ashen goblet Apr 4, 2024, 10:35 PM

#

Nice

#

Out of interest, what did you do for page allocator? Didnt yet look at your repo but I like your project

flint idol Apr 4, 2024, 10:36 PM

#

PMM or VMM?

ashen goblet Apr 4, 2024, 10:36 PM

#

Phys

flint idol Apr 4, 2024, 10:36 PM

#

Freelist

ashen goblet Apr 4, 2024, 10:37 PM

#

I wrote a buddy yesterday may push it

#

Havent tested it performance wise tho

flint idol Apr 5, 2024, 1:20 AM

#

If I ever get audio support on this os, this will be the first thing I play

flint idol Apr 5, 2024, 9:53 PM

#

So, there is no apparent bug with my allocator even with that test + randomized memory

#

So....

#

I really want to implement some sort of way of debugging a panic on real hardware

#

A recovery terminal of sorts on panic

#

To examine memory and stuff

#

tbf that wouldn't be too hard, it's just it would need to be very light, and that'd be hard when I need to check if a keyboard exists

#

specifically a ps/2 keyboard

#

It's too much work that isn't generally worth it

#

I'll make it be a random chance to free a pointer

#

and it hangs somewhere after the 20,000th pass

#

nvm

#

it was just slow

#

I think an optimization is storing the amount of free bytes in a region so the allocator doesn't have to loop every node checking for a free node

#

or the longest continuous node size

flint idol Apr 5, 2024, 11:02 PM

#

Yeah I think this is a lot faster

#

Just what could be a faster way to do this:

SlabNode* longestContinuousNode = nullptr;
for (SlabNode* node = region->freeNodes.tail; node;)
{
    if (!longestContinuousNode || node->size > longestContinuousNode->size)
        longestContinuousNode = node;
    node = node->prev;
}
region->longestContinuousNode = longestContinuousNode;```

#

Of course without sorting the list in ascending order and using the last node

#

Wtf is this stack trace lol

📎 message.txt

#

IT'S 45 KIB

#

This is what I knew would happen if I used an allocator in the VMM and the VMM is the allocator

#

Some random shit like this would happen because the VMM needed to use the slab allocator to allocate a node, which needed to use the VMM to allocate a new region

#

Now I need to make some changes related to the nodes being allocated after the region is set up

#

I think if I just reserve the pages using vmm::Allocate() should setup the nodes in the context.

#

Then I just need to fix the page descriptors

#

to say the truth about the node

near linden Apr 6, 2024, 12:44 AM

#

flint idol Some random shit like this would happen because the VMM needed to use the slab a...

You‘d want to keep some number of descriptors always availabe

flint idol Apr 6, 2024, 12:44 AM

#

I do

#

Kinda

#

Anyway I know how to fix the bug so it's fine

ornate ginkgo Apr 6, 2024, 3:45 AM

#

flint idol Wtf is this stack trace lol

ah yes "450 lines left" 😅

flint idol Apr 6, 2024, 8:30 AM

#

It would be more if the stack were larger or there was no guard page for the stack

flint idol Apr 6, 2024, 8:22 PM

#

I finally got my test laptop working

#

Debugging on real hardware should be easier

#

There was an untested case with SMT on the laptop where the SMP code would do... funny things with the BSP because CPU 0 was used for the hyper threads of each CPU

#

and of course there is a bug where node == nullptr in the slab allocator whilst adding a slab region node

#

AllocateRegionNode is failing...

#

It only fails if the VMM fails to find a free page

#

or if the VMM fails to allocate...

#

In this case, vmm::RawAllocate seems to be failing

#

As the slab allocator object being called is the page node/descriptor allocator

#

Judging by the stack trace

#

vmm::RawAllocate cannot find a base address on it's own, the caller must

#

and in this case, the caller doesn't

#

I just got the 550 line stack trace crash again

#

It was because the AllocateRegionNode() function didn't add the newly region node before trying to reserved the pages

#

This time it should be fixed...

real pecan Apr 6, 2024, 8:42 PM

#

flint idol I finally got my test laptop working

Nice

real pecan Apr 6, 2024, 8:43 PM

#

flint idol I finally got my test laptop working

What about your other hardware

flint idol Apr 6, 2024, 8:45 PM

#

Well there is the other computer I use for testing

#

It's just it's not located close to my dev computer so it's annoying to test on unless I'm testing fixed code

#

or code that should work

#

There is the dev computer

#

Obviously I don't want to restart it on every change to the kernel I make to test it

#

I seem to have fixed the 550 line stack trace

#

Just I still have a crash related to that

#

Slab allocator should be fixed soon

real pecan Apr 6, 2024, 8:49 PM

#

well thats nice

#

you seem to be fixing it quickly

flint idol Apr 6, 2024, 9:48 PM

#

The VMM is being a bitch and not making a page node after the pages are reserved

#

The Allocate() function isn't failing

#

There is no reason that the page node isn't being appended

#

Maybe it was a bug with IsAllocated?

#

if (InRange(node->pageDescriptors[0].virt, 
                    node->pageDescriptors[node->nPageDescriptors - 1].virt + (node->pageDescriptors[node->nPageDescriptors - 1].isHugePage ? OBOS_HUGE_PAGE_SIZE : OBOS_PAGE_SIZE), 
                    base) &&
                   InRange(node->pageDescriptors[0].virt, 
                    node->pageDescriptors[node->nPageDescriptors - 1].virt + (node->pageDescriptors[node->nPageDescriptors - 1].isHugePage ? OBOS_HUGE_PAGE_SIZE : OBOS_PAGE_SIZE), 
                    (uintptr_t)base + size-1))```
I don't think size should be decremented

#

Basically, the node could've been falsely marked as reserved but not committed

#

So Allocate() tried to commit that node

#

That wasn't it

#

Bruh

#

The page node is getting appended

#

According to the print statement

#

So where is it in the context?

#

Conclusion: This bug should also be impossible, yet it happened anyway.

#

github.com/AlDanial/cloc v 1.90  T=3.60 s (30.3 files/s, 3408.9 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C++                             32            345            260           5363
C/C++ Header                    57            486            870           3600
Assembly                         8            124             53            787
CMake                            7             46             23            227
Bourne Shell                     3              4              7             27
Markdown                         1              0              0             25
DOS Batch                        1              4              6             17
-------------------------------------------------------------------------------
SUM:                           109           1009           1219          10046
-------------------------------------------------------------------------------```

#

LOC update

#

114 text files

#

I think this bug is unfixable

#

There is no cause of it

#

This should be impossible

#

But yet it happened anyway

chilly osprey Apr 6, 2024, 10:22 PM

#

flint idol I think this bug is unfixable

i doubt that

flint idol Apr 6, 2024, 10:22 PM

#

Well yeah

#

But it seems to be completely impossible

#

Unless data corruption happened

#

vmm::Allocate calls vmm::Context::AppendPageNode which calls vmm::g_pgNodeAllocator.Allocate

#

Which should have a region since a new one was allocated

#

So that shouldn't fail and return nullptr

#

and if it does there is an assert statement right after that preventing further code execution

#

Then the node is initialized

#

and appended to the list

#

of page nodes

#

Then, later, after the call to vmm::Allocate() to reserve the pages, it tries to get the page node that represents the newly reserved pages

#

Which fails because it's not in the list of nodes

#

Which I was able to confirm by dumping the list of nodes and examining them

#

The address is indeed aligned to 0x1000 so that shouldn't have caused a failure

#

The worst part about this bug is that this code path is only getting triggered on real hardware

#

Rather that or I haven't waited long enough on qemu for it to get triggered

#

Solution to not being able to debug on real hardware: Make a usb driver and gdb stub

#

Ok it just failed in qemu

#

After > 60000 passes

#

I have one chance to get this right

real pecan Apr 6, 2024, 10:40 PM

#

flint idol Solution to not being able to debug on real hardware: Make a usb driver and gdb ...

lol

#

thats gonna be hard

flint idol Apr 6, 2024, 10:40 PM

#

flint idol I have one chance to get this right

before I have to wait a couple minutes once again for 60000 passes to finish

flint idol Apr 6, 2024, 10:41 PM

#

real pecan thats gonna be hard

yeah obviously

#

I'm more worried about the usb driver though if I did do that

real pecan Apr 6, 2024, 10:41 PM

#

flint idol I have one chance to get this right

this opportunity comes once in a lifetime

weary hound Apr 6, 2024, 10:41 PM

#

xhci has been the bane of my existence for a few years now

real pecan Apr 6, 2024, 10:42 PM

#

weary hound xhci has been the bane of my existence for a few years now

i can confirm

weary hound Apr 6, 2024, 10:42 PM

#

now it's not impossible i am just too stupid but it is quite a pain regardless

flint idol Apr 6, 2024, 10:43 PM

#

iwannadie

#

Is why

#

This has been happening

#

(yes I use ddd)

weary hound Apr 6, 2024, 10:43 PM

#

based

flint idol Apr 6, 2024, 10:45 PM

#

In case you couldn't infer what was happening, the page descriptors were freed or uninitialized

#

Most likely the former

real pecan Apr 6, 2024, 10:45 PM

#

flint idol Is why

wtf is this gui

flint idol Apr 6, 2024, 10:45 PM

#

It's ddd on wsl2

#

That's what it is

weary hound Apr 6, 2024, 10:46 PM

#

real pecan wtf is this gui

xmotif goodness

real pecan Apr 6, 2024, 11:06 PM

#

flint idol Is why

Did you figure it out?

flint idol Apr 6, 2024, 11:06 PM

#

Well it was probably because of a free of the page descriptor

#

Otherwise it shouldn't be zeroed out

#

Interesting, the entire region node is zeroed out

#

It is initialized before the page descriptor is allocated

#

It has to have been zeroed out after the page descriptor, otherwise the allocator would've failed in other ways

#

The only thing I can think of is that the same virtual address was allocated before the node was allocated and appended

#

Which is very likely

#

Considering there is nothing stopping that from happening other than the node being present, and the node isn't present

#

I should just raise the IRQL and temporarily stop the other CPUs

#

To prevent that from happening

#

or at the very least raise the IRQL

#

Raising the IRQL doesn't fix it

#

But locking the vmm context might

#

problem is: I don't do that

#

Well, now is the time for me to do that

#

Well, it doesn't crash

#

Just I'm not sure whether it works

#

It seems to be hanging, likely on the spinlock

#

Better than crashing 🤷

#

I added a spin variable that will print a stack trace if a lock has been hanging for too long

#

in the lock function

flint idol Apr 6, 2024, 11:46 PM

#

That didn't fix it

#

nooo

#

I had misordered the reserve call and the Unlock call so it hung

#

That's enough debugging for now

flint idol Apr 7, 2024, 9:23 AM

#

So I did end up doing further debugging, and I figured that the page was (somehow) zeroed in Allocate() while reserving the page

#

I'd need to do more debugging to find out why, although I think it could be because of the page being overwritten when Allocate calls the page node allocator

flint idol Apr 7, 2024, 2:48 PM

#

The bug was because I mixed the protection and flags parameters of vmm::Allocate

real pecan Apr 7, 2024, 2:48 PM

#

Bro

#

If only there were enums

flint idol Apr 7, 2024, 2:49 PM

#

There are

real pecan Apr 7, 2024, 2:49 PM

#

How then

flint idol Apr 7, 2024, 2:49 PM

#

They're integers, right?

real pecan Apr 7, 2024, 2:49 PM

#

If you used enum class the compiler would warn you

chilly osprey Apr 7, 2024, 3:33 PM

#

real pecan If you used enum class the compiler would warn you

wdym

vernal chasm Apr 7, 2024, 3:35 PM

#

chilly osprey wdym

It's not allowed to implicitly cast from an rnum class base type to the enum class or other way

#

You'd need to explicitly do (Enum) v, or (int) v

chilly osprey Apr 7, 2024, 3:36 PM

#

we arent in C but yeah

#

static cast

flint idol Apr 7, 2024, 3:36 PM

#

Yeah but the allocation flags and protection flags are bitfields

vernal chasm Apr 7, 2024, 3:36 PM

#

Don't need static cast for trivial things

flint idol Apr 7, 2024, 3:36 PM

#

So enum class wouldn't help because I'd need to cast

chilly osprey Apr 7, 2024, 3:37 PM

#

vernal chasm Don't need static cast for trivial things

sure you can use C style casts for everything

#

but you can also use printf and malloc and whatnot, doesnt mean you should

vernal chasm Apr 7, 2024, 3:37 PM

#

chilly osprey but you can also use printf and malloc and whatnot, doesnt mean you should

Printf and malloc are better than cout and new lmao

chilly osprey Apr 7, 2024, 3:37 PM

#

not really

flint idol Apr 7, 2024, 3:38 PM

#

printf looks better than cout

vernal chasm Apr 7, 2024, 3:38 PM

#

flint idol So enum class wouldn't help because I'd need to cast

That's why I have a flags type

chilly osprey Apr 7, 2024, 3:38 PM

#

flint idol printf looks better than cout

and it also cant format custom types

vernal chasm Apr 7, 2024, 3:38 PM

#

chilly osprey and it also cant format custom types

It can with glibc

chilly osprey Apr 7, 2024, 3:38 PM

#

anyway std print exists in c++23

chilly osprey Apr 7, 2024, 3:38 PM

#

vernal chasm It can with glibc

i dont consider non standard extensions part of the language

vernal chasm Apr 7, 2024, 3:39 PM

#

Printf is much faster than using cout

chilly osprey Apr 7, 2024, 3:39 PM

#

whats your source for that

vernal chasm Apr 7, 2024, 3:40 PM

#

Cout does sync on every write with <<, meanwhile printf is once

chilly osprey Apr 7, 2024, 3:40 PM

#

so you dont have a source

vernal chasm Apr 7, 2024, 3:41 PM

#

You don't always need a source, when you've tested yourself

chilly osprey Apr 7, 2024, 3:41 PM

#

well show me your results from your tests then

vernal chasm Apr 7, 2024, 3:41 PM

#

Not at home, nor do I keep track of every test I do

chilly osprey Apr 7, 2024, 3:42 PM

#

a quick google search doesnt show the same as what you have said

vernal chasm Apr 7, 2024, 3:42 PM

#

And they tested what?

#

The exact format string matters

chilly osprey Apr 7, 2024, 3:43 PM

#

#

also i dont really see how the performance of printing to the console really matters

vernal chasm Apr 7, 2024, 3:44 PM

#

Now do that with many strings/ints/floats

#

And then tell me what happens when do have multiple threads writing out

chilly osprey Apr 7, 2024, 3:46 PM

#

there are plenty of solutions to that

barren merlin Apr 7, 2024, 3:51 PM

#

vernal chasm Cout does sync on every write with <<, meanwhile printf is once

I thought it was only with std::endl

chilly osprey Apr 7, 2024, 3:53 PM

#

Unless std::ios_base::sync_with_stdio(false) has been issued, it is safe to concurrently access these objects from multiple threads for both formatted and unformatted output.

vernal chasm Apr 7, 2024, 3:54 PM

#

chilly osprey > Unless std::ios_base::sync_with_stdio(false) has been issued, it is safe to co...

That just disables the sync altogether

vernal chasm Apr 7, 2024, 3:54 PM

#

barren merlin I thought it was only with std::endl

It's with each <<
But it flushes its internal buffer on endl

inland radish Apr 7, 2024, 3:55 PM

#

real pecan If you used enum class the compiler would warn you

You can't use those as flags I don't think

real pecan Apr 7, 2024, 3:57 PM

#

u can

#

u can make type operator|(type, type) etc

inland radish Apr 7, 2024, 3:57 PM

#

Example?

chilly osprey Apr 7, 2024, 3:57 PM

#

or just use an unscoped enum

#

and put it in a namespace

real pecan Apr 7, 2024, 3:57 PM

#

chilly osprey or just use an unscoped enum

we just saw an example how that lead to a multiple days of debugging

chilly osprey Apr 7, 2024, 3:58 PM

#

ah did we

#

ive never had issues with it

flint idol Apr 7, 2024, 3:58 PM

#

it was only one day of debugging

#

The other days spent were to try to optimize the allocator

real pecan Apr 7, 2024, 3:58 PM

#

inland radish Example?

enum class flag { one, two };

flag operator|(flag l, flag r) { return static_cast<flag>(static_cast<unsigned>(l) | static_cast<unsigned>(r); }

flint idol Apr 7, 2024, 3:59 PM

#

In fact, wouldn't it error if the invalid type is used

real pecan Apr 7, 2024, 3:59 PM

#

chilly osprey ive never had issues with it

chilly osprey Apr 7, 2024, 3:59 PM

#

what am i looking at here

#

a plane of some kind

real pecan Apr 7, 2024, 3:59 PM

#

you'll figure it out

vernal chasm Apr 7, 2024, 3:59 PM

#

kekw

chilly osprey Apr 7, 2024, 4:00 PM

#

real pecan you'll figure it out

i actually wont

real pecan Apr 7, 2024, 4:00 PM

#

https://en.wikipedia.org/wiki/Survivorship_bias

Survivorship bias

Survivorship bias or survival bias is the logical error of concentrating on entities that passed a selection process while overlooking those that did not. This can lead to incorrect conclusions because of incomplete data.
Survivorship bias is a form of selection bias that can lead to overly optimistic beliefs because multiple failures are over...

chilly osprey Apr 7, 2024, 4:02 PM

#

i dont think i would have gotten that even if i had thought about it

real pecan Apr 7, 2024, 4:02 PM

#

this plane is a pretty common meme so you would've run into it eventually

chilly osprey Apr 7, 2024, 4:03 PM

#

i believe its my first time seeing it

vernal chasm Apr 7, 2024, 4:03 PM

#

real pecan this plane is a pretty common meme so you would've run into it eventually

Yeah, think I can't even count the number of times on my hands and feet xD

flint idol Apr 7, 2024, 4:10 PM

#

https://tenor.com/view/i-saw-what-you-deleted-cat-gif-25407007

Tenor

real pecan Apr 7, 2024, 4:10 PM

#

lmao

vernal chasm Apr 7, 2024, 4:13 PM

#

Oberrow sees everything

flint idol Apr 7, 2024, 4:16 PM

#

Anyway, that's enough allocator testing

#

Time to test uACPI on test subject #3

#

Instant page fault KEKW

#

Definitely a problem with my allocator or something similar

#

As this aml blob has been tested with my previous kernel and worked just fine

#

Nvm

#

A similar crash happens on qemu

#

Still a problem with my allocator though

real pecan Apr 7, 2024, 4:20 PM

#

i thought u fixed it KEKW

flint idol Apr 7, 2024, 4:20 PM

#

Well

#

I did

#

Then another bug came

#

or maybe it's something else

#

I'll test on my previous kernel with the current uACPI version

real pecan Apr 7, 2024, 4:22 PM

#

do u assume its a uacpi regression lol

flint idol Apr 7, 2024, 4:23 PM

#

real pecan Apr 7, 2024, 4:23 PM

#

i mean could be but ridiculously unlikely

#

i mean it dies in qemu lol

flint idol Apr 7, 2024, 4:27 PM

#

nvm, I can't test on my previous kernel

#

I don't want to make the new kernel api again

real pecan Apr 7, 2024, 4:28 PM

#

if anything i've fixed tons of fuzzing issues recently

flint idol Apr 7, 2024, 4:32 PM

#

All I know is that the allocator is giving a pointer that exceeds the boundaries of a region

#

Thus causing a page fault

#

Instead of finding the root cause, I'll just make the allocator detect this happening, and make it try allocating from the next region

real pecan Apr 7, 2024, 4:33 PM

#

flint idol All I know is that the allocator is giving a pointer that exceeds the boundaries...

like the allocation function is broken?

flint idol Apr 7, 2024, 4:33 PM

#

Yeah basically

#

The end of the pointer given > region end

real pecan Apr 7, 2024, 4:34 PM

#

cant u track down why

flint idol Apr 7, 2024, 4:34 PM

#

Well for that I'd need to be able to break at the exact allocation that does that

#

As it doesn't happen for each allocation

lean glen Apr 7, 2024, 4:37 PM

#

just use liballoc at this point

real pecan Apr 7, 2024, 4:39 PM

#

fr

#

that could at least help you eliminate your allocator bugs

flint idol Apr 7, 2024, 4:39 PM

#

At this point, I might as well use my old's kernel allocator

lean glen Apr 7, 2024, 4:43 PM

#

I mean ideally you'd just debug your allocator properly

#

in userspace

#

and testing it by replacing it in e.g gcc

real pecan Apr 7, 2024, 4:44 PM

#

lean glen I mean ideally you'd just debug your allocator properly

#1141057599584878645 message

lean glen Apr 7, 2024, 4:45 PM

#

lol

flint idol Apr 7, 2024, 4:47 PM

#

What in uACPI is making a 2144 byte allocation?

#

Is my question

real pecan Apr 7, 2024, 4:56 PM

#

Who knows

weary hound Apr 7, 2024, 5:06 PM

#

flint idol What in uACPI is making a 2144 byte allocation?

attach a debugger and add a breakpoint

flint idol Apr 7, 2024, 5:46 PM

#

Ok I fixed the bug

#

It was a node's size being incorrectly calculated at initialization

#

Welp, the allocator failed on one of uACPI's requested

#

With failed meaning: returned nullptr to signify error

#

That should be impossible though

#

Allocate() should never return nullptr

#

I'll log what it returns in uacpi_kernel_alloc

#

uacpi_namespace_load: aborting execution due to previous error: out of memory
the allocation before that is 0x39b0 bytes in size

#

anyway I'll be back in an hour to figure out why this fails

flint idol Apr 7, 2024, 6:48 PM

#

Well idk what happened, but now it's hanging somewhere

#

idk where

#

idk if it's in the allocator, while taking a lock

#

All I know is that it hangs somewhere

real pecan Apr 7, 2024, 6:49 PM

#

is gdb not allowed?

flint idol Apr 7, 2024, 6:49 PM

#

Not on real hardware

#

Unfortunately

#

Is there a way to make uACPI trace a lot more than the usual UACPI_LOG_TRACE option

real pecan Apr 7, 2024, 6:50 PM

#

yea

#

in the interpreter comment out #define UACPI_TRACE_OP or whatever its called

#

it will log every opcode thats gets executed

flint idol Apr 7, 2024, 7:01 PM

#

Well, now I know it hangs somewhere on the execution of a FieldOp

#

Looking at the interpreter code for processing FieldOp, there seem to be no calls to kernel api

real pecan Apr 7, 2024, 7:14 PM

#

They happen in the microcode

#

But add logs if unsure

#

The code you're looking at processes an already parsed opcode

flint idol Apr 7, 2024, 7:26 PM

#

I now got a bunch of print statements in most kernel api

#

Time to wait for the opcode to come up again

#

Why the hell is there a single opcode that has multiple pages of logging solely for allocations

#

It's not even done logging the allocations it's doing

#

Maybe that was the FieldOp

real pecan Apr 7, 2024, 7:29 PM

#

Fieldop is one of the more complex ops

flint idol Apr 7, 2024, 7:29 PM

#

Because there still hasn't been another opcode since then

real pecan Apr 7, 2024, 7:29 PM

#

It has a variable list of named objects that must be created

flint idol Apr 7, 2024, 7:29 PM

#

Yeah it was the FieldOp because it just hung

#

An allocation of 12752 bytes

real pecan Apr 7, 2024, 7:30 PM

#

Unlucky

flint idol Apr 7, 2024, 7:30 PM

#

Is making the allocator hang

#

Possibly because of all the regions there are

#

Good thing I have a function called OptimizeAllocator which will hopefully do something here

real pecan Apr 7, 2024, 7:30 PM

#

Lol

#

So why does it hang exactly

flint idol Apr 7, 2024, 7:31 PM

#

So the allocator has something called region nodes

#

They represent a region of memory

#

There are probably so much, that in trying to find a free one, the allocator takes a stupidly long amount of time to find one

#

A lot of them might have no allocated nodes at all

#

OptimizeAllocator defragments the region nodes and removes any regions with no allocated nodes

#

I'll just make uacpi_kernel_alloc call it every few hundred allocations

real pecan Apr 7, 2024, 7:36 PM

#

Not a very good design lol

flint idol Apr 7, 2024, 7:36 PM

#

it'll be fine

#

I made the spin lock print something if it spins for a long time

#

and of course it's in OptimizeAllocator

#

recursive lock

#

oops

#

At this point if the allocator is causing so much trouble

#

I should rewrite it or use a different one

ashen goblet Apr 7, 2024, 7:45 PM

#

probably

real pecan Apr 7, 2024, 7:45 PM

#

definitely, yes

flint idol Apr 7, 2024, 7:46 PM

#

welp, time to nuke the allocator

#

This allocator will be nothing but git history now

#

Time to welcome back the old kernel's allocator

#

But heavily ported to fit this kernel's allocator structure

#

tbh I should've done this a lot earilier

#

Like before the scheduler

flint idol Apr 7, 2024, 8:34 PM

#

I've ported the allocator

#

I just need to do one more thing before it works

#

I did that thing

#

Time to test it

#

Immediate page fault

#

lol

#

Anyway time to debug

real pecan Apr 7, 2024, 8:51 PM

#

lol

#

Your kernel might be cursed

flint idol Apr 7, 2024, 8:58 PM

#

hopefully not

#

It was because of some dependency bugs

chilly osprey Apr 7, 2024, 8:59 PM

#

i wish i could say my kernel isnt cursed

flint idol Apr 7, 2024, 8:59 PM

#

new depended on the allocator being initialized

#

and new was called before the allocator was initialized

#

the return value of new is not checked once

#

so it page faulted trying to access the zero page

#

Some parts of my kernel are cursed because of some hacks to make inter-dependendent parts of the kernel work

#

Those parts are extremely fragile and can break if you modify them

#

Example: VMM initialization

#

Now instead of getting a page fault I get this:

Stack trace:
        0xffffffff8000d4bb: Cannot get function name
        0xffffffff8000d3fe: Cannot get function name
        0xffffffff80010676: Cannot get function name
        0xffffffff80010993: Cannot get function name
        0xffffffff800102ba: Cannot get function name
        0xffffffff80010348: Cannot get function name
        0xffffffff8001150a: Cannot get function name
        0xffffffff80000465: Cannot get function name
        0x0000000000000000: Cannot get function name```

#

Which is good

#

Still doesn't work but better than page fault

#

It was because of the same reason as this #1141057599584878645 message

#

I fixed that and it hangs

real pecan Apr 7, 2024, 9:07 PM

#

Why does it hang

flint idol Apr 7, 2024, 9:07 PM

#

I'm trying to figure that out

real pecan Apr 7, 2024, 9:08 PM

#

Does it hang in qemu?

flint idol Apr 7, 2024, 9:08 PM

#

It was hanging in the allocator

#

not uACPI

#

Should've mentioned that

#

This is the same allocator as the previous kernel

#

The one that worked with uACPI

#

On all of my avaliable ASL blobs

#

it was hanging because the allocator was locked and vmm::Initialize was trying to call it

#

Recursive locking = bad

real pecan Apr 7, 2024, 9:11 PM

#

Does it not hang anymore

flint idol Apr 7, 2024, 9:11 PM

#

Where?

#

In uACPI?

real pecan Apr 7, 2024, 9:11 PM

#

In general

flint idol Apr 7, 2024, 9:11 PM

#

Well I need to fix that bug

#

Then I'll check

#

Hopefully not

real pecan Apr 7, 2024, 9:13 PM

#

Inb4 quadruple fault

flint idol Apr 7, 2024, 9:15 PM

#

Nah, instead a seemingly impossible code path was taken

#

nvm it's not impossible

#

// There must be something seriously wrong with this page block if this if statement is executed.
if (!current->firstBlock || !current->lastBlock)
{
    current = current->next;
    if (!current)
        break;
    if (!current->firstBlock && !current->lastBlock)
        freePageBlock(current->prev);
    continue;
}```

#

That if statement happened

#

To fix that, I just need to remove the if statement

#

It thinks that block is corrupted

#

Which it would be in normal cases

#

except this isn't a normal case

#

I fixed the hanging

#

Somehow something forgets to unlock the allocator

#

nvm

#

Just an oversight

#

now it should work

#

still hangs

real pecan Apr 7, 2024, 9:22 PM

#

Bruh

flint idol Apr 7, 2024, 9:22 PM

#

I'll just stop yapping in here for every bug I find and just say when I fix it

#

that should speed stuff up

real pecan Apr 7, 2024, 9:23 PM

#

Nah its fun to read

flint idol Apr 7, 2024, 9:23 PM

#

hold on, you enjoy this lol

real pecan Apr 7, 2024, 9:23 PM

#

Yes

#

Allocator bugs are fun

flint idol Apr 7, 2024, 9:25 PM

#

it hangs in the demand page fault handler

#

Rather, it gets in an infinite loop because of it

#

I just realized at that point it doesn't exist

#

this time it should work

#

bruh

#

division error

#

But it at least got farther than vmm initialization

#

Function uacpi_kernel_free, line 318: free of object by uACPI not allocated by uACPI allocator. This is a bug, please report in some way.```

#

Welp, time to fix that

#

~~didn't get that with previous allocator, time to rollback~~

#

Just got this on reboot:

Function Allocate, File /mnt/c/Code/obos/src/oboskrnl/allocators/liballoc.cpp, Line 212: Assertion failed, "currentPageBlock->highestBlock->magic == MEMBLOCK_MAGIC". Kernel heap corruption detected for block 0xffff8000fee020a0, allocAddr: 0x0000000000000000, sizeBlock: 0x0000000000000000!```

#

Just rebooted and it disappeared

real pecan Apr 7, 2024, 9:34 PM

#

Unlucky

flint idol Apr 7, 2024, 9:54 PM

#

flint idol ``` Function uacpi_kernel_free, line 318: free of object by uACPI not allocated ...

Memory corruption nooo

real pecan Apr 7, 2024, 9:55 PM

#

real pecan Your kernel might be cursed

.

flint idol Apr 7, 2024, 9:55 PM

#

I just it hopefully isn't

#

I never said it isn't

#

just hopefully

#

isn't

real pecan Apr 7, 2024, 9:56 PM

#

Lol

flint idol Apr 8, 2024, 1:21 PM

#

I decided to change the allocation method for the allocator to use a free list instead of basically bump off of the highest block

real pecan Apr 8, 2024, 1:25 PM

#

inb4 instant page fault

flint idol Apr 8, 2024, 1:26 PM

#

That is what happened at first, rather that or a hang

#

I fixed those bugs, and it got further into uACPI

#

Before page faulting again

#

because the allocator provided an invalidly sized memory block

#

0x20 bytes were requested and the pointer given was 0x10 bytes to a page boundary to a page that was unmapped

#

I should probably add guard pages to these page regions

ashen goblet Apr 8, 2024, 1:36 PM

#

At which point will you admit that you should have debugged and tested your allocator in userspace? halfmemeright

flint idol Apr 8, 2024, 1:36 PM

#

this is the old kernel's allocator

#

the one that was supposed to work

#

and it did in the old kernel

#

and it is now

ashen goblet Apr 8, 2024, 1:37 PM

#

Big ouch

flint idol Apr 8, 2024, 1:37 PM

#

I just need to fix it...

#

It wasn't the allocator's fault

#

It was the kernel api's fault

#

in this case

ashen goblet Apr 8, 2024, 1:39 PM

#

That sounds like an easy fix then luckily

flint idol Apr 8, 2024, 1:39 PM

#

it did some wacky stuff to the size parameter

#

that it only needed to do with the other allocator

flint idol Apr 8, 2024, 1:55 PM

#

I tried to fix a bug with the allocator

#

got a triple fault

#

how fun

flint idol Apr 8, 2024, 2:12 PM

#

how does

block->next = nullptr;
block->prev = nullptr;```
Correlate to a page fault happening in the thread context switch function?

#

I think my kernel is cursed

#

My kernel is cursed.

#

~~time for rewrite 5~~

#

jk

#

I did something else, and now rather qemu bails out or uacpi magically makes qemu shutdown

#

qemu bails out

#

**
ERROR:../../../system/cpus.c:504:qemu_mutex_lock_iothread_impl: assertion failed: (!qemu_mutex_iothread_locked())
Bail out! ERROR:../../../system/cpus.c:504:qemu_mutex_lock_iothread_impl: assertion failed: (!qemu_mutex_iothread_locked())

#

KEKW

#

A rebuild should fix it...

#

it doesn't

#

It's because the cpu_local pointer contains the IRQL, but it somehow wasn't demand paged it seems

#

nvm

#

it might be because of that

real pecan Apr 8, 2024, 2:31 PM

#

flint idol ~~time for rewrite 5~~

Lol

flint idol Apr 8, 2024, 2:40 PM

#

The triple fault is during the gdt initialization

#

*cpu-local gdt initialization

#

the gdt is null except for the tss

#

Maybe this is why KEKW

memzero(g_cpuInfo, sizeof(cpu_local) * g_nCPUs);

#

Now it triple faults somewhere else

#

Seems as if the hhdm page tables are corrupted

#

How do I mess up that badly

#

More than the hhdm page tables

#

It triple faults around the free of an object

#

I wonder if I tested vmm::Free thonk

flint idol Apr 8, 2024, 3:26 PM

#

It's a bug with FreePageTables() thonk

#

hmmm

#

idk how but I freed the stack by accident

#

idk how it's still running after that, probably the TLB providing mappings

#

More than the stack in fact, everything after the HHDM

#

up until the kernel code

#

*framebuffer

flint idol Apr 8, 2024, 3:37 PM

#

flint idol It's a bug with `FreePageTables()` <:thonk:575445129863888896>

*FreePageMapAt

#

Why does this bug have to come up now, idk

#

But time to fix it

#

well this is the only time that function gets to be called

flint idol Apr 8, 2024, 9:09 PM

#

I just love it when the VMM fails on an allocation of 8 pages with a whole-ass 48-bit address space

blk = (pageBlock*)vmm::Allocate(
    &vmm::g_kernelContext,
    nullptr,
    nPages * OBOS_PAGE_SIZE, 
    vmm::FLAGS_GUARD_PAGE_LEFT|vmm::FLAGS_GUARD_PAGE_RIGHT,
    0
);```

#

with all the parameters valid

#

out of physical memory should panic so that isn't the case

#

I'll run qemu with 1g instead of 512mib

#

same issue

#

thinkong

#

The new allocator makes real hardware suddenly not work anymore

#

LiPerformDelisquesceUponObjectW

flint idol Apr 8, 2024, 9:22 PM

#

flint idol The new allocator makes real hardware suddenly not work anymore

or well the old allocator

#

why is the page block 0xffffffffffffffff

#

what

#

maybe I forget to set something to zero

real pecan Apr 8, 2024, 9:24 PM

#

the os being cursed is not even a joke anymore

flint idol Apr 8, 2024, 9:25 PM

#

maybe a rewrite of some parts, or the entire thing would be sufficient to fix it

real pecan Apr 8, 2024, 9:25 PM

#

lol

flint idol Apr 8, 2024, 9:26 PM

#

Like I always say, the most cursed kernel I've written is the second kernel

#

It was unstable

#

and the code organization is 💩

#

and it being unstable was unstable

real pecan Apr 8, 2024, 9:27 PM

#

so what happened with this rewrite

flint idol Apr 8, 2024, 9:27 PM

#

well

#

it started out uncursed

#

then uACPI came in

#

and made everything go to shit

#

not uACPI's fault

real pecan Apr 8, 2024, 9:27 PM

#

lol

#

i think u sped run it a bit

flint idol Apr 8, 2024, 9:28 PM

#

but everything went to shit after intergrating it

flint idol Apr 8, 2024, 9:28 PM

#

real pecan i think u sped run it a bit

nah it's good allocator testing KEKW

#

I'd rather have this now than in the middle of writing the driver interface

#

or even worse, vfs

#

#1141057599584878645 message

#

If you thought this was speedran, the previous kernel got the scheduler within the first week

real pecan Apr 8, 2024, 9:31 PM

#

lol

flint idol Apr 8, 2024, 9:33 PM

#

and it's just the allocator that's cursed at this point

#

everything else behaves

#

the allocator is like that one weird kid in a kindergarten classroom

#

they all behave except him

#

solution: rewrite the ~~kernel~~ allocator

#

I tried porting the one that worked

#

Barely worked

#

the one before that barely worked after days of debugging

#

but before I do that

#

I need to make sure I can't fix this bug

#

I already ported it, I don't want to be left with an even shittier allocator if the rewrite doesn't work

real pecan Apr 8, 2024, 9:37 PM

#

ye

flint idol Apr 8, 2024, 9:38 PM

#

and if none of them work, liballoc is my saviour

ashen goblet Apr 8, 2024, 9:40 PM

#

My favorite progress report

flint idol Apr 8, 2024, 9:41 PM

#

yeah none of the others have any detailed bug reports (because they don't have many bugs like I do)

#

or if they do there's are boring bugs

#

If I just make this allocator simple af, there should be no bugs, right?

#

Like if there is no room to fail, it shouldn't fail

#

Like if free just slaps a node to the free list

#

and allocate removes a node from the free list to the allocated list

#

and returns a pointer to that node's data

#

everyone's happy

#

reallocate just does some memcpying and freeing and allocation

real pecan Apr 8, 2024, 9:54 PM

#

imagine needing realloc

flint idol Apr 8, 2024, 9:55 PM

#

I don't think it's used anywhere

real pecan Apr 8, 2024, 9:55 PM

#

the entire linux code base has 138 usages

flint idol Apr 8, 2024, 9:55 PM

#

I just implement it just in case

#

There was a dynamic array class I was working on

#

Idk what happened to that

ashen goblet Apr 8, 2024, 9:55 PM

#

real pecan the entire linux code base has 138 usages

Well thats just the file count isnt it

#

Some files may use it multiple times

real pecan Apr 8, 2024, 9:56 PM

#

ashen goblet Well thats just the file count isnt it

most files are one time use, some are 2 times

#

i also see one 3 time and one 4 time

devout niche Apr 8, 2024, 10:44 PM

#

try testing your allocators in userspace

#

i did

real pecan Apr 8, 2024, 10:45 PM

#

devout niche try testing your allocators in userspace

#1141057599584878645 message

#

You're the third person to recommend him that

devout niche Apr 8, 2024, 10:46 PM

#

real pecan You're the third person to recommend him that

third time lucky

real pecan Apr 8, 2024, 10:46 PM

#

KEKW

flint idol Apr 8, 2024, 10:58 PM

#

ok fine I will

flint idol Apr 9, 2024, 1:55 AM

#

I just finished writing the allocator, I'll test it tomorrow

flint idol Apr 9, 2024, 9:19 PM

#

I opened the project I had for testing things in visual studio, and to my surprise, I had an allocator

#

That I had tested

#

It seems to have been an allocator I was making for my second kernel

#

but ended up ditching because it was 💩

flint idol Apr 9, 2024, 9:45 PM

#

I tested the allocator with allocations from 1-2048 over 100,000 passes

#

there were a couple small bugs that I fixed

flint idol Apr 9, 2024, 9:45 PM

#

flint idol I tested the allocator with allocations from 1-2048 over 100,000 passes

in userspace this time

#

All the passes were done in under a minute

#

I'll run it with optimizations

#

nah 💀

#

msvc compiles out my allocator test function

real pecan Apr 9, 2024, 9:47 PM

#

sooo

#

time to port to obos

flint idol Apr 9, 2024, 9:47 PM

#

I may or may not have written it in obos

#

and ported it over to windows

#

I just want to see how it works with optimizations

#

and with the rng seeded

#

it works

#

without problem

real pecan Apr 9, 2024, 9:49 PM

#

will you finally uncurse your kernel

flint idol Apr 9, 2024, 9:49 PM

#

If I figure out how it is other than the allocator, then yes

#

Probably the VMM

#

The VMM and allocator being interdependant probably wasn't a good idea

#

I've applied the changes

#

Time to test

#

Well, it works (kinda)

#

except for the fact the uacpi causes the allocator to fail massively

#

with a very, very, very long stack trace

real pecan Apr 9, 2024, 9:54 PM

#

lol

flint idol Apr 9, 2024, 9:55 PM

#

📎 message.txt

#

635 lines 💀

#

so uhh very stable kernel I got here

ashen goblet Apr 9, 2024, 9:56 PM

#

I thought you were gonna uncurse it

#

You made it worse

flint idol Apr 9, 2024, 9:57 PM

#

I think what I just did should fix it

#

nope

#

time to examine the stack trace

#

I'll test the allocator in user-space again but with a bigger range

#

for the size

#

uACPI allocates big objects

#

Lesson learnt: do not allocate one million passes of an allocator with the max size being 4 pages

#

unless you don't mind casually using 4 gib of memory

ashen goblet Apr 9, 2024, 10:03 PM

#

Your allocators high limit is 4 pages?

flint idol Apr 9, 2024, 10:03 PM

#

no

#

I just decided 4 pages max for the allocator tests

#

when choosing the random number

ashen goblet Apr 9, 2024, 10:04 PM

#

oh mb

flint idol Apr 9, 2024, 10:04 PM

#

It also rounds up sizes to a 4 page boundary when allocating a page region

real pecan Apr 9, 2024, 10:05 PM

#

Maximum allocation size for uACPI should be a few pages

#

Unless its some crazy recursive aml and stuff

flint idol Apr 9, 2024, 10:06 PM

#

A lot of memory is intentionally being leaked, so I'll fix that

#

Every three allocations, a block is freed

#

It works in userspace

#

even with bigger allocations

#

sooooooooooooooooooooooooooooooooooooooooooooooooooooo

#

I'll run the tests in kernel-mode

flint idol Apr 9, 2024, 10:59 PM

#

100,000 passes while testing the allocator finish successfully

#

and then uACPI faults?

#

in both kernel-mode and user-mode

weary hound Apr 9, 2024, 11:00 PM

#

wonder if uacpi triggers kasan in managarm

flint idol Apr 9, 2024, 11:00 PM

#

prob not

#

it was the allocator during uACPI that faulted

#

I mean you could always check

flint idol Apr 9, 2024, 11:31 PM

#

A region node has an invalid next pointer

#

It points to a dead block it seems

#

Which causes a page fault

#

irdk how

#

As that should be impossible

#

I'll examine the logs of freeing and allocations from uACPI

#

and since I made logging only go to port 0xe9 the logs are a lot faster

#

for some reason uACPI frees a lot of objects at the end of something which I think is weird

#

Mainly because uACPI mainly allocates

#

It's borrowing my resources and never giving them back >:(

limber wave Apr 10, 2024, 12:23 AM

#

After reading this thread, I realized that allocators are a real tricky thing to deal with

flint idol Apr 10, 2024, 12:59 AM

#

nope, just a skill issue on my side

#

mainly with my vmm

#

there is a weird interdependance

#

that was causing problems

#

bugs with allocators suck

#

to deal with

real pecan Apr 10, 2024, 6:34 AM

#

flint idol for some reason uACPI frees a lot of objects at the end of something which I thi...

nah that's exactly how it works

#

all per-op items are freed at the end

vernal chasm Apr 10, 2024, 6:35 AM

#

real pecan all per-op items are freed at the end

Do you also use a GC as well?

real pecan Apr 10, 2024, 6:50 AM

#

vernal chasm Do you also use a GC as well?

no ofc not

#

that would be a huge overkill

#

objects are reference counted

vernal chasm Apr 10, 2024, 6:59 AM

#

real pecan objects are reference counted

Ah ok

flint idol Apr 10, 2024, 12:29 PM

#

I probably won't do much development on this today

flint idol Apr 10, 2024, 5:11 PM

#

After fixing the allocator, I feel like making a makeshift debugger inside the kernel

#

Won't be anything fancy

#

Just some step, examine, and register viewing commands

#

and exception handling

#

It would take a bit, but it'll be worth it for debugging real hardware

flint idol Apr 10, 2024, 5:13 PM

#

flint idol Just some step, examine, and register viewing commands

and possibly breakpoints

#

The only thing I can explain this bug with now is page table corruption

#

The region that wasn't mapped causing a page fault was never freed

#

unless...

#

The region before it in was freed

#

maybe somehow the region size was invalid

#

invalid as in too big

#

so it freed both

#

regions

#

It was a bug in the code for rounding up

#

🤦‍♂️

#

more specifically a typo

#

size += (size - (size % pageSize));

#

should've been

size += (pageSize - (size % pageSize));

#

Function allocateNewRegion, File /mnt/c/Code/obos/src/oboskrnl/allocators/basic_allocator.cpp, Line 308: Assertion failed, "res". Could not reserve page block.```

#

Now I got that

#OBOS (not vibecoded)