OBOS (not vibecoded) | Operating System Development | Page 5

ashen goblet Apr 10, 2024, 5:40 PM

#

Just write a rounding macro smh

limber wave Apr 10, 2024, 5:40 PM

#

Have you looked at what allocators other OSes use?

flint idol Apr 10, 2024, 5:41 PM

#

I used liballoc at one point

limber wave Apr 10, 2024, 5:41 PM

#

Idk, like what allocation systems use professonal OSs

flint idol Apr 10, 2024, 5:41 PM

#

my previous kernel's allocator was based on it

#

and my current allocator is based on my previous kernel's allocator

#

oh no

#

pages are getting reallocated

#

when they already are allocated

#

This of course has to only happen now

#

I'll just set the kernel off with a million passes of the allocator

#

I should probably make the test function multithreaded

#

anyway I gtg now though

#

110,000 passes doesn't fault or anything

#

or break

#

I should just uncurse my VMM

#

It's the part that's cursed

#

the rest of the kernel is fine

#

the allocator isn't the problem, the VMM is

#

well both were the problem at one point

real pecan Apr 10, 2024, 5:52 PM

#

This is what happens when instead of making a PAGE_ROUND_UP helper you write the same logic ad hoc in multiple places

#

You're bound to get it wrong at one point

limber wave Apr 10, 2024, 5:54 PM

#

flint idol I used liballoc at one point

Is it a Linux allocator?

flint idol Apr 10, 2024, 5:54 PM

#

no

#

https://discord.com/channels/440442961147199490/1170856016666177727

#

that

flint idol Apr 11, 2024, 12:56 AM

#

Why must everything I fix cause another bug

Page fault in kernel-mode at 0xffffffff8000e7f2 (cpu 0, pid 0, tid 0) while trying to read a non-present page. The address of this page is 0x0000000000000006. IRQL: 0. Error code: 0. whileInScheduler = false
Dumping registers:
        RDI: 0xffffffff800471c0, RSI: 0xffff8000fee00000, RBP: 0xffff8000fee47b58
        RSP: 0xffff8000fee47af8, RBX: 0xfffffffffffff000, RDX: 0x0000000000000000
        RCX: 0x0000000000007000, RAX: 0x0000000000000006, RIP: 0xffffffff8000e7f2
         R8: 0x0000000000000000,  R9: 0x0000000000000000, R10: 0x0000000000000000
        R11: 0x0000000000000000, R12: 0xffff8000fee4795c, R13: 0x0000000000000000
        R14: 0xffff8000fee4795c, R15: 0x0000000000000000, RFL: 0x0000000000200206
         SS: 0x0000000000000010,  DS: 0x0000000000000010,  CS: 0x0000000000000008
        CR0: 0x0000000080010015, CR2: 0x0000000000000006, CR3: 0x000000000009e000
        CR4: 0x0000000000040620, CR8: 0x0000000000000000, EFER: 0x0000000000000d00```

#

I fix one bug

#

That comes up from the VMM

#

The only thing I can think that could cause that is:
allocator returning bogus address (unlikely, 0x6 isn't aligned to 0x10)
memory corruption

rapid pulsar Apr 11, 2024, 12:59 AM

#

♫ 99 little bugs in the code ♫
♫ 99 little bugs ♫
♫ take one down, patch it around ♫
♫ 137 little bugs in the code ♫

flint idol Apr 11, 2024, 1:00 AM

#

(trust me, I'll fix them eventually)

#

Rather that, or I'll start rewrite 5

#

This time, with a working VMM™

#

I have a feeling something isn't respecting the red zone not existing

#

ahem ahem libgcc ahem ahem

#

and that's causing stack corruption

#

Causing the address accessed to be 0x6

flint idol Apr 11, 2024, 1:25 AM

#

Welp, that wasn't the problem

#

At least I fixed it before it became the problem

empty kernel Apr 11, 2024, 3:49 AM

#

flint idol *ahem ahem* libgcc *ahem ahem*

I use a special build of libgcc that doesn't use the red zone. I think you just need to modify one config file in libgcc before building.

flint idol Apr 11, 2024, 11:10 AM

#

I'm going to run the kernel's build with -fanalyzer

#

and see how the compiler yells at me

#

It found a bug with my intergration of stb sprintf

#

and with stb printf itself

#

use of uninitialized values

#

while calling the callback to print

flint idol Apr 11, 2024, 11:42 AM

#

All I'm getting is useless diagnostics about new() returning nullptr

flint idol Apr 11, 2024, 1:10 PM

#

flint idol The only thing I can think that could cause that is: allocator returning bogus a...

It seems like memory corruption

#

In the VMM

#

How fun

flint idol Apr 11, 2024, 5:38 PM

#

So, I got pretty bored of debugging that bug

#

and the allocator does work when not under heavy load (uACPI)

#

So I decided to start making the kernel debugger

flint idol Apr 11, 2024, 7:20 PM

#

So, it almost can give pong from the ping command

real pecan Apr 11, 2024, 7:21 PM

#

lmao

flint idol Apr 11, 2024, 7:25 PM

#

now it doesn't page fault, just it still doesn't work

real pecan Apr 11, 2024, 7:27 PM

#

is that like a gdb stub or something?

flint idol Apr 11, 2024, 7:27 PM

#

gdb stub is too much work

#

it's going to be a very basic debugger in the kernel

#

#1141057599584878645 message

real pecan Apr 11, 2024, 7:29 PM

#

i can promise u in the end you will end up doing more work

flint idol Apr 11, 2024, 7:30 PM

#

except my hardware doesn't have any serial port of any sort, and I don't want to make a nic/usb driver for the sole purpose of running a gdb stub

real pecan Apr 11, 2024, 7:37 PM

#

well

#

a usb driver with basic serial-over-usb is probably less work than a full blown kernel debugger

#

and you will need usb code anyway

flint idol Apr 11, 2024, 7:41 PM

#

the ping command works

#

does anyone know how to make netcat not echo the characters I type

#

ok I got it to do that

flint idol Apr 11, 2024, 8:21 PM

#

I got two commands done, help and ping

#

I know very useful commands

#

2/11 done

real pecan Apr 11, 2024, 8:22 PM

#

how will you implement this

flint idol Apr 11, 2024, 8:22 PM

#

x86 debug registers

real pecan Apr 11, 2024, 8:22 PM

#

like the breakpoint for example

flint idol Apr 11, 2024, 8:22 PM

#

duh

real pecan Apr 11, 2024, 8:22 PM

#

would be simpler to just replace the first byte with int3

flint idol Apr 11, 2024, 8:23 PM

#

Well, I'd need to change the protection of the page first

#

so I can write to it

#

that's probably more complicated then just using a hardware breakpoint

#

or querying if a breakpoint was hit on every debug exception (assuming EFLAGS.TRAP is set)

real pecan Apr 11, 2024, 8:24 PM

#

well if it works it works

flint idol Apr 11, 2024, 8:25 PM

#

step is the next simplest, so I'll just implement that

real pecan Apr 11, 2024, 8:25 PM

#

yeah but steps.. what exactly

flint idol Apr 11, 2024, 8:25 PM

#

instruction

real pecan Apr 11, 2024, 8:25 PM

#

where

flint idol Apr 11, 2024, 8:25 PM

#

step
Steps one instruction

flint idol Apr 11, 2024, 8:25 PM

#

real pecan where

one instruction forward?

real pecan Apr 11, 2024, 8:26 PM

#

which out of 100 kernel threads

flint idol Apr 11, 2024, 8:26 PM

#

current

real pecan Apr 11, 2024, 8:26 PM

#

how do u set current

flint idol Apr 11, 2024, 8:26 PM

#

https://tenor.com/view/invincible-that's-the-neat-part-you-don't-gif-14180388429135427516

Tenor

real pecan Apr 11, 2024, 8:26 PM

#

lol

flint idol Apr 11, 2024, 8:27 PM

#

I said simple kernel debugger

#

enough to be able to see rbp on panic or something

#

Currently, debugging on real hardware is printf

#

After this, it'll be fancy printf

#

and getline

flint idol Apr 11, 2024, 8:30 PM

#

flint idol Currently, debugging on real hardware is printf

and if a panic happens, it's printf+me recording the output of the kernel until the panic happens so I know where the panic originated

real pecan Apr 11, 2024, 8:30 PM

#

tbh instruction stepping is not useful

#

only source line stepping is useful

flint idol Apr 11, 2024, 8:31 PM

#

Well I have no other choice unless I want to delve into the world of dwarf

#

(I don't)

real pecan Apr 11, 2024, 8:31 PM

#

might as well do it to have a fancy debugger

#

getting a source line is not complex

flint idol Apr 11, 2024, 8:32 PM

#

I have a slight feeling you're telling me it's not complex so I try to do it and end up ditching this and making a gdb stub

real pecan Apr 11, 2024, 8:33 PM

#

lol

flint idol Apr 11, 2024, 8:41 PM

#

I've implemented step and finish, time to test them

#

They will most definitely fail first try

#

But I might as well

#

step worked first try

#

at least the first step command did

#

step works

#

I just thought of a bug with finish

#

If it just goes to the next ret

#

the function will probably call something else

#

Making finish go to another function's ret

#

It's a pretty simple fix though

#

I made a very, very, very, minimal disassembler for the finish command.
It supports disassembing the entire x86 call and ret instructions (without identifying any operands)

#

Basically if it sees a call instruction before a finish command finishes, it increments a counter, and when it sees a return instruction, it decrements that count. If there is a ret instruction, and the count is zero, then the finish command is done executing

#

Well it almost works

#

I think

#

Yeah it doesn't work

#

Well step still works so I at least didn't break that

#

I fixed

#

it

#

I got finish and step working

#

What next

#

I guess continue

#

That's pretty simple

#

Just one line of code

#

or it should be at least

#

dbg_state.shouldStopAtNextInst = false;

#

Well it works

#

except it's slow af

#

solution: idk

#

I can't solve it by disabling the trap flag because of how breakpoints will work

#

or I could just change how breakpoints work

#

oh wait I messed up

#

I made the debugger state global

#

and not cpu-local

#

whoopsie

#

or thread-local

#

I'll make it cpu-local

#

in case I need to debug the scheduler or something

#

some parts of the state should be global, such as breakpoint info

#

others should be cpu-local

flint idol Apr 11, 2024, 9:23 PM

#

flint idol or I could just change how breakpoints work

or I could use the scheduler in this and make it process breakpoints, but that wouldn't be very reliable because the scheduler works on every timer interrupt and not instruction

#

I'll need to make breakpoints use hardware breakpoints

#

#1141057599584878645 message

#

instead of that

#

step has a "small" bug

#

#

It steps a "bit" more than one instruction

#

It was a bug where the trap flag wasn't set

#

well it was set

#

but never restored in the actual context

#

if that makes sense

#

I will now implement dump registers

#

it's the simplest out of all commands

#

just a printf statement

#

Done:

#

I think I'll implement the examine memory commands

#

I'll copy my strtoull function from my previous kernel's uACPI implementation to decode those

devout niche Apr 11, 2024, 9:58 PM

#

what's in scope for this grand kernel debugger?

#

i could use with one myself

flint idol Apr 11, 2024, 9:58 PM

#

#1141057599584878645 message

devout niche Apr 11, 2024, 9:58 PM

#

underrated and traditional feature

flint idol Apr 11, 2024, 9:58 PM

#

Nothing complicated at all

#

Just useful enough to debug real hardware without wanting to shoot yourself

#

also it's c++

#

It is kind of easy-to-port

#

As long as you have printf and some other misc stuff

#

RaiseIRQL/LowerIRQL

#

a way to get into exception handlers

#

ie. isr handlers

hybrid kraken Apr 11, 2024, 10:29 PM

#

Support WinDbg meme

flint idol Apr 11, 2024, 10:34 PM

#

lol

flint idol Apr 11, 2024, 11:13 PM

#

x/dec and x/hex work

#

I think

#

it does indeed work

#

that leaves me with:

break
break_at
print

flint idol Apr 11, 2024, 11:15 PM

#

real pecan i can promise u in the end you will end up doing more work

@real pecan What happened to this?

#

Anyway, breakpoints will be global, not cpu-local. They will use the x86 debug registers to work because of reasons stated above

#

But before I get to that, I'll implement print

#

There is a bug with x/* that I accidentally just discovered

#

If there is one argument instead of the two needed

#

it prints all it can from the address until it decides to page fault

#

If there are no arguments passed it just page faults

#

I might also want to implement a command for setting registers and writing memory

#

That would be useful

flint idol Apr 12, 2024, 12:08 AM

#

I got the print command done

#

That only leaves me with the breakpoint commands

#

and the writing commands

#

ie. writing to memory and registers

#

I added a command to set the value of a register

#

I am now going to implement 4 commands for writing memory:
wb (write byte)
ww (write word)
wd (write dword)
wq (write qword)

#

Then I will implement breakpoints

#

Then I need to test whether the ps/2 keyboard interface works

#

(I added two ways to input commands, rather from serial, or from the keyboard)

#

Then I can consider the kernel debugger complete for now

#

of course after testing on real hardware

#

I might eventually make it a bit more complex

#

Right now, it's sitting at 618 lines

#

not including io code

#

that's an additional 326 lines

flint idol Apr 12, 2024, 12:53 AM

#

I implemented wb, ww, wd, and wq

#

Time to implement breakpoints

flint idol Apr 12, 2024, 2:16 AM

#

I just implemented breakpoints

#

I just need to do two more things with them

#

I just need to implement function name->rip conversion

#

It'll be simple enough

#

I just need to loop through the symbol table until I find the function name and I got the rip

#

All that's left for me to do after that is to test this on real hardware with the PS2 keyboard

#

Interesting...

#

While using the makeshift ps/2 keyboard driver

#

I'll fix it tomorrow

real pecan Apr 12, 2024, 7:22 AM

#

flint idol <@353593948523986945> What happened to this?

wdym

flint idol Apr 12, 2024, 9:21 AM

#

real pecan wdym

You said this would be more work than writing a GDB stub

#

and now I'm almost done

real pecan Apr 12, 2024, 9:46 AM

#

gdb stub can do source line stepping

#

and arbitrary struct views

#

you can only do insn level stepping which isnt useful

flint idol Apr 12, 2024, 9:47 AM

#

True

#

If I wanted that I would probably have ended up writing a gdbstub

real pecan Apr 12, 2024, 9:48 AM

#

yes

flint idol Apr 12, 2024, 9:48 AM

#

But this is more for examining memory and system state on crash

#

or before a crash

#

In real hardware

#

On qemu I'd just use the gdbstub for that

flint idol Apr 12, 2024, 10:43 AM

#

I fixed the bug with the makeshift ps/2 keyboard driver

#

and now the debugger should be testable on real hardware

real pecan Apr 12, 2024, 11:53 AM

#

Does it work?

flint idol Apr 12, 2024, 11:53 AM

#

It indeed does

#

I just implemented break with ~~function~~ symbol names

#

Bruh

#

My power just went out

#

ok it's back

#

I got all commands implemented, but it still feels a bit incomplete

#

I think I'll add a disassembly

#

and stack trace

#

I'll first add a stack trace because that's easier

#

Then I'll use a library I used in my 2nd kernel that I used to disassemble

#

"ZyDis"

#

I got a basic stack trace command

#

All that's left is the disassembler

flint idol Apr 12, 2024, 1:25 PM

#

CMake was being a bitch and now I need to rewrite part of my CMakeLists.txt

#

No way

#

It DELETED some of my new files

#

(example: everything since the last commit)

#

How does this happen?

#

What fucking sequence of events caused this to happen?

#

Who asked CMake to git reset --hard HEAD?

vernal chasm Apr 12, 2024, 1:44 PM

#

flint idol I got a basic stack trace command

You're implementing gdb without gdb?

flint idol Apr 12, 2024, 1:57 PM

#

Well I'm implementing it

#

then testing it

#

and if it doesn't work I use gdb

#

then I fix it

#

Anyway, I got some files to rewrite/restore

#

Like don't you just love it when cmake git reset --hard HEADs everything

#

for no ~~apparent~~ reason

#

🫠

#

One of the the only files that I hoped was still in the editor, kdbg.cpp, the file that has the actual debugger, wasn't in the editor

#

So it's lost forever

#

iwannadie

flint idol Apr 12, 2024, 2:03 PM

#

flint idol CMake was being a bitch and now I need to rewrite part of my CMakeLists.txt

This happened while writing the dependency file for ZyDis

#

I would've lost my new kernel allocator if it weren't for the fact that I had it saved in another file because of the tests I made

real pecan Apr 12, 2024, 2:04 PM

#

Clion saves snapshots every time btw

flint idol Apr 12, 2024, 2:04 PM

#

I'm using notepad++ lol

real pecan Apr 12, 2024, 2:04 PM

#

It has saved me once because of accidental hard reset a few weeks worth of work

flint idol Apr 12, 2024, 2:04 PM

#

notepad++ isn't that complicated

#

but I'm hoping it's complicated enough to have that

#

the place where they'd be stored, %APPDATA%\Notepad++\backup\, has nothing

#

Well, if I don't want to rewrite the debugger (I don't), I could use Ghidra to decompile the debugger's functions from the usb

#

I do have io.cpp and init.cpp

#

I'm just lacking the debugger part

flint idol Apr 12, 2024, 2:09 PM

#

flint idol Well, if I don't want to rewrite the debugger (I don't), I could use Ghidra to d...

then I'll refactor the decompiled functions

#

Before I do that I need to rewrite some of the headers

#

and port back the allocator

#

I'm missing three files from the kernel debugger - 2 headers and one source file

#

Then I'm not missing anything else I can remember, except I need to reapply some bug fixes to the VMM and stuff

#

Mainly VMM bugs looking at the thread

#

The allocator I wrote had some bugs which I fixed but didn't document here

#

So I'll have fun rediscovering those

#

If I'm lucky, I might have debug symbols I can use to more accurately decompile the kernel

#

But I strip the binary before putting it into the isodir

#

and before I started messing with the dependencies, I had cleaned the build tree

flint idol Apr 12, 2024, 2:25 PM

#

flint idol If I'm lucky, I might have debug symbols I can use to more accurately decompile ...

I do however have symbol names

#

So I don't need to suffer and refind those

#

Ghidra doesn't understand else if, so I'm going to use another decompiler

#

Thank god I didn't use templates here

#

Not that I know a place where I might

#

but still

flint idol Apr 12, 2024, 3:35 PM

#

Yeah I ain't decompiling this

#

Time to rewrite the kernel debugger ugh

#

WAIT

#

I FOUND A STASH WITH THE KENRELE FFDEBUGGETR

#

*KERNEL DEBUGGER

#

FUCK YEAH

#

I GOT THE KERNEL DEBUGGER AND OTHER CHANGES BACK

#

Time to commit this so this doesn't happen again

#

also I'm backing it up here

📎 kdbg.zip

#

The code is pushed for safety mesures

#

for future reference, if something ever accidentally git revert --hard HEADs your code, you can use git fsck --lost-found to find any lost files

#

Just if you ran git gc between then and the git fsck command, you have to hope the changes were stashed somewhere

flint idol Apr 12, 2024, 5:07 PM

#

I got disassembly working

#

(credits to ZyDis for the disassembler)

#

I got the syntax upside-down

#

#

I fixed the syntax to be x/i size [addr]

#

so that the size can be optional

chilly osprey Apr 12, 2024, 5:22 PM

#

why are you writing a disassembler in your kernel?

flint idol Apr 12, 2024, 5:22 PM

#

debugger

#

also I didn't write it

#

it's a library

chilly osprey Apr 12, 2024, 5:23 PM

#

sure

#

my question still stands

#

why

flint idol Apr 12, 2024, 5:23 PM

#

The debugger can be disabled in the kernel options

flint idol Apr 12, 2024, 5:23 PM

#

chilly osprey why

because debugging isn't very convenient without source, so this is the most I could do

#

add the disassembly

chilly osprey Apr 12, 2024, 5:24 PM

#

flint idol because debugging isn't very convenient without source, so this is the most I co...

do you not have access to your own source code?

flint idol Apr 12, 2024, 5:24 PM

#

yeah, but I don't want to addr2line everything

chilly osprey Apr 12, 2024, 5:25 PM

#

why not use gdb?

flint idol Apr 12, 2024, 5:25 PM

#

#1141057599584878645 message

#

For debugging real hardware

chilly osprey Apr 12, 2024, 5:26 PM

#

just implement the protocol

flint idol Apr 12, 2024, 5:26 PM

#

None of my hardware has a serial port

#

#1141057599584878645 message

flint idol Apr 12, 2024, 5:28 PM

#

real pecan how do u set current

I mean technically you can use the write commands to overwrite debugger state

#

and then you can choose the thread

real pecan Apr 12, 2024, 5:28 PM

#

chilly osprey just implement the protocol

I already told him he doesn't care lmao

chilly osprey Apr 12, 2024, 5:28 PM

#

real pecan a usb driver with basic serial-over-usb is probably less work than a full blown ...

note this

chilly osprey Apr 12, 2024, 5:28 PM

#

real pecan I already told him he doesn't care lmao

🤷‍♂️

flint idol Apr 12, 2024, 5:28 PM

#

This isn't supposed to be fully blown

devout niche Apr 12, 2024, 5:29 PM

#

having a kernel debugger is a good and accepted tradition, the fathers of many modern kernels incorporated one

flint idol Apr 12, 2024, 5:29 PM

#

Just enough™ to debug real hardware without printf

#

Yeah see Fadanoid gets it

devout niche Apr 12, 2024, 5:30 PM

#

linux is very unusual in not having one (the reason is because linux torvalds thought it made kernel dev too easy)

real pecan Apr 12, 2024, 5:31 PM

#

flint idol Yeah see Fadanoid gets it

Fadanoid also supports Motorola 6800

devout niche Apr 12, 2024, 5:31 PM

#

real pecan Fadanoid also supports Motorola 6800

you are fake news

#

i only support the 68000

real pecan Apr 12, 2024, 5:32 PM

#

devout niche linux is very unusual in not having one (the reason is because linux torvalds th...

Well windows only has kernel api stub implemented

#

And the actual debugging is done via a windbg app

real pecan Apr 12, 2024, 5:32 PM

#

devout niche i only support the 68000

Damn

devout niche Apr 12, 2024, 5:33 PM

#

real pecan Well windows only has kernel api stub implemented

i thought it had a bit more implemented in-kernel as the famous Kd

real pecan Apr 12, 2024, 5:34 PM

#

Tbh maybe im not that familiar with how its implemented

devout niche Apr 12, 2024, 5:35 PM

#

real pecan Tbh maybe im not that familiar with how its implemented

nor me so i don't want to give out so much rope that i hang myself with it, but it would fit in with what's been traditional in kernels

flint idol Apr 12, 2024, 5:35 PM

#

real pecan Well windows only has kernel api stub implemented

Windows also has drivers for usb, NICs, whatever communication device you can name. I don't, and I don't want to support those at the moment. If I had a USB driver, I probably would write a gdb stub instead, as gdb > than whatever I can come up with in terms of ease of use

real pecan Apr 12, 2024, 5:37 PM

#

Yes but you will need USB eventually

#

So right now you're wasting time

flint idol Apr 12, 2024, 5:37 PM

#

I was also wasting time implementing uACPI so early and not getting onto the driver interface

#

and it took around the same time to complete the kernel api

chilly osprey Apr 12, 2024, 5:38 PM

#

do you not ever need uacpi?

real pecan Apr 12, 2024, 5:38 PM

#

Because you wanted to implement the driver interface in terms of it soo

real pecan Apr 12, 2024, 5:38 PM

#

flint idol and it took around the same time to complete the kernel api

Yes but its not throwaway code

flint idol Apr 12, 2024, 5:38 PM

#

Neither is this

devout niche Apr 12, 2024, 5:38 PM

#

i love to use acpi to get irq and i/o port assignments for ps/2 keyboards and such

flint idol Apr 12, 2024, 5:38 PM

#

(I hope)

devout niche Apr 12, 2024, 5:38 PM

#

so much nicer than hard-coding

chilly osprey Apr 12, 2024, 5:38 PM

#

but surely once you have a network driver you will use gdb

#

or usb

flint idol Apr 12, 2024, 5:39 PM

#

depends on three things:

If I ever implement those
If I remember by then that I should implement a gdb stub for the kernel
If I need a kernel debugger for real hardware by then

#

Now that I'm done the kernel debugger though, I can go back to figuring out why the hell there is memory corruption in the VMM when I try to initialize uACPI

real pecan Apr 12, 2024, 7:31 PM

#

flint idol Now that I'm done the kernel debugger though, I can go back to figuring out why ...

any luck

flint idol Apr 12, 2024, 9:52 PM

#

Well, something is zeroing a node in the allocator

#

Which would be easy enough to find, except the allocator only zeroes in two places, ReAllocate when newSize < nodeSize and on most Allocate calls

#

nvm it doesn't in Allocate

#

ReAllocate is only called in the kernel debugger's getline function

#

I'll disable it to see if ReAllocate is the problem

#

or if it's something else

#

It isn't a problem with ReAllocate

#

or at least not with the kernel debugger being present

real pecan Apr 12, 2024, 10:07 PM

#

you know what would be easier?

flint idol Apr 12, 2024, 10:07 PM

#

what

real pecan Apr 12, 2024, 10:07 PM

#

dump the tables from your laptop

#

then feed them to QEMU

flint idol Apr 12, 2024, 10:08 PM

#

oh I'm debugging on qemu right now

real pecan Apr 12, 2024, 10:08 PM

#

if that particular table causes your allocator to choke it will repro

real pecan Apr 12, 2024, 10:08 PM

#

flint idol oh I'm debugging on qemu right now

lol

#

it dies on qemu now as well?

flint idol Apr 12, 2024, 10:08 PM

#

Since I made this new allocator it did

real pecan Apr 12, 2024, 10:08 PM

#

i see

flint idol Apr 12, 2024, 10:10 PM

#

I feel like making a >= become > will fix it

#

just at random

#

nope

real pecan Apr 12, 2024, 10:11 PM

#

flint idol I feel like making a `>=` become `>` will fix it

..or maybe actually read the code and figure out whats wrong

flint idol Apr 12, 2024, 10:11 PM

#

I am

#

Nothing is apparently wrong in the Allocate() function

#

So there shouldn't be a reason for the node of an object to be zeroed

#

at this point, I should use uACPI in user mode with Qemu's blob and my kernel's allocator

real pecan Apr 12, 2024, 10:13 PM

#

put a hw breakpoint?

#

to that node

#

to see where it gets zeroed

flint idol Apr 12, 2024, 10:13 PM

#

oh yeah I forgot that exists

real pecan Apr 12, 2024, 10:13 PM

#

flint idol at this point, I should use uACPI in user mode with Qemu's blob and my kernel's ...

yup

flint idol Apr 12, 2024, 10:14 PM

#

I just need to add that functionality to my kernel debugger, as it only supports code breakpoints

#

I'll just leave that as todo

real pecan Apr 12, 2024, 10:14 PM

#

flint idol at this point, I should use uACPI in user mode with Qemu's blob and my kernel's ...

this is the better idea

flint idol Apr 12, 2024, 10:14 PM

#

and put in a hardware breakpoint

#

I assume runner/CMakeLists.txt is the one I use for building uACPI's tests

real pecan Apr 12, 2024, 10:16 PM

#

yes

#

interface_impl.cpp

#

is waht u want to look at

flint idol Apr 12, 2024, 10:38 PM

#

KEKW

#

it happened as soon as I ran it

real pecan Apr 12, 2024, 10:38 PM

#

lmao

#

where did u even get the table?

flint idol Apr 12, 2024, 10:38 PM

#

acpidump -b

real pecan Apr 12, 2024, 10:38 PM

#

well should be trivial to debug now

flint idol Apr 12, 2024, 10:40 PM

#

Well that occurred because uACPI gave me a nullptr object to free

real pecan Apr 12, 2024, 10:41 PM

#

..did u not have a check for that?

flint idol Apr 12, 2024, 10:41 PM

#

Where, in the allocator

#

or in the kernel api

real pecan Apr 12, 2024, 10:41 PM

#

somewhere

flint idol Apr 12, 2024, 10:41 PM

#

Yeah I do

#

In QueryObjectSize, which is used in this case to find whether an object is allocated or not, it fails automatically if !obj

real pecan Apr 12, 2024, 10:42 PM

#

ok good

#

does it get further now

flint idol Apr 12, 2024, 10:42 PM

#

Nope

#

It can't be the allocator if uACPI gives a null object

#

I've set breakpoints on the uacpi_kernel_alloc and uacpi_kernel_calloc if the value returned is nullptr

#

they weren't hit

real pecan Apr 12, 2024, 10:43 PM

#

wait

#

did u not know that free(nullptr) is well defined?

#

to be a no op

flint idol Apr 12, 2024, 10:44 PM

#

no

real pecan Apr 12, 2024, 10:44 PM

#

its used everywhere in uacpi

flint idol Apr 12, 2024, 10:44 PM

#

bruh

real pecan Apr 12, 2024, 10:44 PM

#

delete in c++ supports nullptr as well

#

how did u not know lol

flint idol Apr 12, 2024, 10:49 PM

#

It doesn't fail with my dev computer's acpi tables

#

How do I get qemu's blob?

real pecan Apr 12, 2024, 10:49 PM

#

dont tell me this entire time its been failing for u because of missing nullptr check

flint idol Apr 12, 2024, 10:49 PM

#

nope

#

only in the usermode test

#

in the kernel, it checks for nullptr and returns

chilly osprey Apr 12, 2024, 10:49 PM

#

real pecan dont tell me this entire time its been failing for u because of missing nullptr ...

ive had huge issues that fall down to tiny things like that

weary hound Apr 12, 2024, 10:50 PM

#

flint idol How do I get qemu's blob?

one way is to just boot archiso or something, install acpica-utils, and run acpidump

real pecan Apr 12, 2024, 10:50 PM

#

^ basically this

#

qemu AML is generated at runtime

weary hound Apr 12, 2024, 10:50 PM

#

for device trees qemu offers -M dump-dtb=filename but i'm not aware of anything like that for acpi

real pecan Apr 12, 2024, 10:50 PM

#

yeah there isn't one

flint idol Apr 12, 2024, 10:51 PM

#

weary hound one way is to just boot archiso or something, install acpica-utils, and run acpi...

how do I get the blob back onto the host

real pecan Apr 12, 2024, 10:51 PM

#

mount

#

send to yourself via discord/telegram

#

many ways

#

qr code

flint idol Apr 12, 2024, 10:52 PM

#

or what I do is dump the table in my kernel onto the serial port, and then have netcat piping into a file that will take the dsdt

real pecan Apr 12, 2024, 10:53 PM

#

sure

weary hound Apr 12, 2024, 10:53 PM

#

-drive file=fat:rw:some/directory/,format=raw,media=disk and mount the disk somewhere in the guest

chilly osprey Apr 12, 2024, 10:58 PM

#

real pecan qr code

carrier pigeon

real pecan Apr 12, 2024, 10:58 PM

#

lol

flint idol Apr 12, 2024, 11:54 PM

#

unexpected error: DSDT declares that it's bigger than dsdt.dat```
![thinkong](https://cdn.discordapp.com/emojis/815589724424241152.webp?size=128 "thinkong")

#

oh wait, I need to output the SDT as well

#

With the qemu's DSDT, it works in userspace

#

Time for hardware breakpoint

flint idol Apr 13, 2024, 12:25 AM

#

I put in a hardware breakpoint, and when it returns, it gives me stack corruption

#

The breakpoint was triggered once while the node was being allocated

#

and idk about anything past that because of the stack corruption

#

examining the region node at the time of the watchpoint being triggered, it's all zeroed

#

Which is.. interesting

#

(that should be impossible in this case, because it was already added to the list of regions of the allocator)

#

and it was being zeroed via a call to memzero, which only happens when the allocator object is that of the VMM

#

and it isn't; it's the general kernel allocator

#

Now I know something is trying to memzero one full page

#

at the region's address

#

I think it was here:

memzero((void*)newPd.virt, newPd.isHugePage ? OBOS_HUGE_PAGE_SIZE : OBOS_PAGE_SIZE);

for demand paging

#

Which explains:
a) why the stack is corrupted
b) why it's so out of the blue
c) why the allocator was still locked

#

but that still makes no sense

#

this is put at the end of Allocate:

if (freeNode == (void*)0xffff8000fee5e340)
    setRWWatchpoint(freeNode);

#

The demand page handler would've been called by then

#

thinkong

#

It seems as if the region gets freed

#

and then gets reallocated

#

so the demand page handler gets called

#

which makes sense looking at the logs

#

Just I need to confirm that actually ever happens

#

For future reference: debugging an allocator with a kernel debugger that uses the allocator you're trying to debug will be a bit unstable

#

I've decided I'm going to switch back to visual studio for developing the kernel. Even though visual studio and non-ms tools are like me with girls; they repel, it's honestly a lot more convenient than having to switch between three applications to write code (notepad++), build (wsl terminal), and debug (ddd)

flint idol Apr 13, 2024, 1:15 AM

#

flint idol It seems as if the region gets freed

Well, that wasn't the case

#

The problem was vmm::Allocate returned the same value twice in a row

#

ie. reallocated an already existing page

#

The only reason I think this could happen assuming my FindBase function isn't broken is memory corruption in the vmm allocator

#

the vmm push node function needs to push nodes in order

#

so it does some logic

#

that logic fails here

#

and slaps the page node at the end of the page node list

#

making the list out of order

#

causing FindBase to shit itself

#

https://github.com/oberrow/obos/blob/driver-interface/src/oboskrnl/vmm/pg_context.cpp#L29

flint idol Apr 13, 2024, 1:41 AM

#

flint idol The problem was vmm::Allocate returned the same value twice in a row

That should be impossible, it changes if the address that it's trying to map can be allocated before mapping anything

#

and failing if it can't

#

I think the bug was a missing i - 1

#

in vmm::Free

#

endPd = i;

#

which caused the node to never be removed from the list of free nodes

#

causing the logic of the push node function to append the node at the end of the list even though that's not where it goes

#

causing FindBase to shit itself

#

Now I get this

#

#

This is another reason I wrote the kernel debugger

#

So I don't have to attach a debugger on cases like this

#

and so I can just get some quick info about the crash

#

not as a fully-fledged, source level, debugger, such as gdb

#

also gdb won't stop at cpu exceptions unless I tell it to

#

or kernel panics

#

and gdb can't stack trace through interrupt frames unless I do some goofy stuff in the assembly

#

I think that was because of a % in my PRI*64 macros

#

for uACPI

#

#

It finally works again

#

time to test on real hardware

#

It works

#

All this because of a missing - 1

#

Test subject one works

#

I shall test on dev computer (test subject 2)

#

then test subject 3

#

Test subject 2 doesn't receive keyboard input so I can't make the kernel debugger continue

#

I'll try test subject three

#

It works

#

I also saw some OSI calls in trace

#

Or whatever it was called

#

it also found some PCI root nodes

#

Idk why but it says 8 processors on a 4-core machine without hyper threading

#

2 thermal zones

#

And 135 devices

#

Now that my kernel is uncursed, I can continue on with the driver interface

#

Which I am hoping to start tomorrow

#

#

Because this screen doesn't show my reflection (I don't want you guys to know how I look), I sent a screenshot of the kernel

#

I'd send more screenshots but test subjects one and two have screens that show my reflection

#

The firmware asks if Linux is supported

#

Which it isn't because uACPI is based and works like the NT interpreter

weary hound Apr 13, 2024, 2:29 AM

#

not even linux says it's linux

flint idol Apr 13, 2024, 2:30 AM

#

What does linux say it is then?

weary hound Apr 13, 2024, 2:30 AM

#

windows

#

mainly because firmwares that check for linux usually have bugs in the linux codepaths

flint idol Apr 13, 2024, 2:31 AM

#

Yeah God forbid firmware developers test on anything but windows

#

Like a wise osdever said, firmware developers are the scum of the earth

#

#bootloaders message

inland radish Apr 13, 2024, 8:28 AM

#

flint idol Because this screen doesn't show my reflection (I don't want you guys to know ho...

I don't want you guys to know how I look
We don't really care tbh

real pecan Apr 13, 2024, 8:43 AM

#

flint idol Idk why but it says 8 processors on a 4-core machine without hyper threading

That's just namespace devices, that don't have to match the number of CPUs that you have

#

But congrats on fixing it

flint idol Apr 13, 2024, 7:29 PM

#

Just pushed the bug fix

#

Time to work on the driver interface

#

if anyone want's qemu's dsdt, here it is:

📎 dsdt.dat

flint idol Apr 13, 2024, 8:05 PM

#

@real pecan Regarding uacpi_kernel_install_interrupt_handler, is the irq parameter an interrupt vector number, or is it an IRQ for the IOAPIC or PIC? Looking the parameter passed to it from uacpi_initialize_events (FADT.SciInt), which in the ACPI documentation is said to be a vector for the PIC or the Global System Interrupt for the IOAPIC, I'm assuming the latter, an IRQ for the IOAPIC/PIC is what's passed.

flint idol Apr 13, 2024, 11:08 PM

#

I've started writing the driver loader

#

After that, I'm writing device discovery

flint idol Apr 14, 2024, 1:31 AM

#

The bulk of the loader is done

#

Mainly because I copied so much code from the previous kernel

#

I copied and pasted all relocation code

#

All that's left for the driver loader is processing the header and stuff

flint idol Apr 14, 2024, 2:59 AM

#

I got a test "driver" working

#

#include <int.h>
//#include <klog.h>

#include <driver_interface/header.h>

using namespace obos;
[[maybe_unused]] volatile
driverInterface::driverHeader __attribute__((section(OBOS_DRIVER_HEADER_SECTION))) g_driverHeader = {
    .magic = driverInterface::g_driverHeaderMagic,
    .type  = driverInterface::driverType::KernelExtension,
    .friendlyName = "Test driver",
    .requestedLoader = {0},
    .loaderPacket = nullptr
};

namespace obos
{
    namespace logger
    {
        __attribute__((weak)) size_t log(const char* format, ...);
    }
}

extern "C"
void _start()
{
    (g_driverHeader);
    logger::log("In test driver!\n");
    while (1);
}```

#

I still need to finalize the driver loader function

#

But I'll commit now

real pecan Apr 14, 2024, 7:58 AM

#

flint idol <@353593948523986945> Regarding `uacpi_kernel_install_interrupt_handler`, is the...

Its the legacy number

flint idol Apr 14, 2024, 3:12 PM

#

I've finalized the driver load function

#

To set up a driver id and stuff

#

and add it to a list of drivers

#

I also added a separate function to start drivers

#

You'd call LoadDriver first with the driver's file data, then you'd use the id that the function returned to start the driver

#

driverInterface::StartDriver(driverInterface::LoadDriver(driverData, driverSize)->id);```

flint idol Apr 14, 2024, 4:43 PM

#

Just pushed the code for the driver loader

#

I can now start on the device discovery code

#

But it kind of depends on there being a VFS

#

https://github.com/oberrow/obos/blob/driver-interface/src/oboskrnl/driver_interface/header.h#L71

#

I also added some documentation for the driver header

#

I think I'll start the VFS today

#

and merge the driver interface branch

#

when I'm done the VFS, I'll make a branch for device discovery

#

and by then, I should be able to start user mode stuff

#

like syscalls

#

then idk what I'll do

#

probably more userspace shit

real pecan Apr 14, 2024, 4:47 PM

#

Nice

flint idol Apr 14, 2024, 4:47 PM

#

whoops

#

I accidentally pushed code that doesn't compile

#

the kernel is sitting at 13400 loc

flint idol Apr 14, 2024, 8:46 PM

#

I just realized the names of the namespace that most functions/classes in most files in the kernel source match one of the parent directories' name

#

example: all functions in the vmm directory are in namespace obos::vmm

#

Kind of how I'm pretty sure NT organizes all functions in the kernel under a directory where the function prefix is. Like KeLowerIrql would be under $kernel_source_root/Ke

#

Except they did that purposefully, and I did it by accident

#

Anyway I should really get to my VFS

flint idol Apr 15, 2024, 1:56 AM

#

Accurate description of me coding my kernel

flint idol Apr 15, 2024, 2:15 AM

#

So, the driver loader only works on test subject one

#

Test subject three page faults in VerifyDriver

#

Test subject two panics

#

But it deadlocks in the printf locks

#

So idk what happened

#

And test subject 2's kernel debugger hangs in the ps/2 keyboard makeshift driver

#

So I had to disable it

#

Then when I went to test subject three I still had it disabled

#

So idk what happened there

inland radish Apr 15, 2024, 6:14 AM

#

flint idol Accurate description of me coding my kernel

Recompiling without changing any code actually DOES fix things sometimes

#

For example there may be an issue with the environment's header dependency tracking so it mistakenly compiles different versions of structs in a header for different compile units

flint idol Apr 15, 2024, 10:30 PM

#

so for some reason the kernel's build system isn't working on linux because path stuff

#

so I gotta fix that

flint idol Apr 15, 2024, 11:07 PM

#

I fixed that

#

Now I'm developing on my laptop to fix the makeshift PS/2 keyboard driver bug

#

So I added some logging to the initialization code

#

and it hangs while waiting for the write buffer to be empty

#

in the ps/2 controller

#

weird part is that this code was copied straight from my previous kernel

#

and the ps/2 keyboard worked there

#

just found a bug with the scrolling code?

#

for the kernel's console

#

It's a memcpy?

#

How does that fail?

#

other than page fault/gpf

flint idol Apr 15, 2024, 11:46 PM

#

Conclusion: The ps/2 keyboard emulation on test subject two sucks

flint idol Apr 17, 2024, 12:53 AM

#

I've added a contribution guide for the OS (not that anyone will, but I might as well)

#

https://github.com/oberrow/obos/blob/master/contribution.md

#

btw no rust is allowed

#

or no pull request merge for you

#

I also added branch protection rules to master

#

So no more direct pushes to master

#

or github will yell at you

real pecan Apr 17, 2024, 9:19 AM

#

flint idol So no more direct pushes to master

i find it useful to sometimes force push if i accidentally merge a pull request and forget to change something

flint idol Apr 19, 2024, 11:28 PM

#

I like how I made a branch called driver-interface-fixes, and continued to not modify the driver interface in there at all KEKW

#

anyway, I did some work on the VFS

#

(design)

flint idol Apr 20, 2024, 1:23 AM

#

oh wow VFS design is a lot more fun than debugging the vmm and allocator

#

and copy+pasting driver code

#

trust me the vfs will be a lot cooler this time

#

I'm not necessarily proud of the previous one KEKW

#

It'll be a node graph (which is how unix does chooses mount points) VFS

#

I hope

#

Something like that

#

I'll have something related to block devices

#

So mount points don't have to be referring to partitions

#

on disk

#

Block device registering and stuff will be the job of the driver interface though

flint idol Apr 20, 2024, 1:29 AM

#

flint idol It'll be a node graph (which is how unix does chooses mount points) VFS

While the VFS will use the node graph concept from unix, it won't be doing stuff the same way as unix. For example, index nodes will have a different meaning than on unix VFSes

#

Index nodes represent a file's data

#

the index node also contains user permission info

flint idol Apr 20, 2024, 1:30 AM

#

flint idol Index nodes represent a file's data

or a mount point

#

maybe I won't do that...

#

actually nvm I will

#

it makes enough sense

#

they're basically directory entries

#

but they also represent the file more directly

#

actually I think this may be unix VFS but with different names to structs

#

I'll read up on VFS design before going further

#

I might want to implement async io eventually

#

because cool

#

#filesystems message

#

but that might be dettering me a bit

#

(I've also never used async io personally)

flint idol Apr 20, 2024, 1:42 AM

#

flint idol or a mount point

hmmm seems interesting how sunos did mount points

#

I think I'll do it like that

#

Basically, if an index node represents a mount point, its children will be kept intact, but the data will be pointing at another inode where the search is continued

#

actually no it'll still point at a struct mpoint

#

but the struct will contain the children

#

where traversal will begin

#

my mount point structure looks like this:

struct mpoint
{
    struct block_device* dev;
    index_node_list root;
};```

#

which should be sufficient

#

except I'll add a field to say what index node represents it

#

struct mpoint
{
    struct block_device* dev;
    index_node_list root;
    index_node* representative;
};```

#

block_device will be left as a todo (for now)

flint idol Apr 20, 2024, 2:03 AM

#

my inodes seem to be vnodes

flint idol Apr 20, 2024, 2:55 AM

#

On the weekend I will work on opening, reading, and maybe writing files\

#

then mounting

#

I'll make a makeshift ramdisk-like filesystem as the filesystem it'll be operating on

#

it won't even be a filesystem

#

it'll be some if statements basically

#

anyway sounds like a problem for future me

unkempt pawn Apr 20, 2024, 11:35 AM

#

jesus theres a lot of messages in this thread lol

ashen goblet Apr 20, 2024, 11:58 AM

#

It may have the most out of any thread here

steep ravine Apr 20, 2024, 12:48 PM

#

keyronex is on par and im pretty sure mintia had more but it seems to me that the thread is gone nowadays

#

but u gotta respect the dedication

opaque pulsar Apr 20, 2024, 1:21 PM

#

Time to spam in the Managarm thread meme

flint idol Apr 20, 2024, 1:23 PM

#

steep ravine keyronex is on par and im pretty sure mintia had more but it seems to me that th...

Mintia has more, and it's gone because hyenasky was banned

#

A lot of the messages here might as well be me debugging shit

flint idol Apr 21, 2024, 5:51 PM

#

I just fixed a bug relating to the kernel not knowing the framebuffer is mapped. It wasn't ever triggered, but it's a good thing I caught it before it would've gotten triggered.

#

Basically the kernel never recognized the framebuffer being mapped in the kernel context

#

Making it possible for the kernel to unknowingly map the framebuffer region, even though it's already mapped.

#

I should probably add some debug-only sanity checks in the map function

#

If I did have those, the two weeks of debugging for it to turn out to be a VMM bug would've been a lot shorter

#

Assertion failed: !isMapped(addr, size)
Is a lot more descriptive than random memory corruption and free bugs

flint idol Apr 21, 2024, 7:02 PM

#

flint idol `Assertion failed: !isMapped(addr, size)` Is a lot more descriptive than random ...

I just added that

#

and first thing I get

#

is

#

Function Allocate, File C:/Code/obos/src/oboskrnl/vmm/map.cpp, Line 478: Assertion failed, "!pd.present". Attempt to reallocate address ffff8000fee00000.```

#

as you can see, I have very stable code

#

I got other things I gotta do

#

So I'll fix this later

#

oh wait

#

I think it's because of the hack I used in the allocator

#

because it has an interdependency on the vmm

#

aight I fixed it

flint idol Apr 21, 2024, 9:38 PM

#

I compiled the kernel with optimizations

#

Shits itself while starting the other processors

#

ie. it triple faults

#

It gpfs in the smp trampoline on the segment 0x8

#

ie. the kernel code segment

#

It seems to be before switching to long mode

#

actually

#

while it's switching to long mode

#

while reloading the descriptors

#

anyway, enough guess work, more jmp $

#

adding jmp $ makes the kernel continue onto initializing the scheduler?

#

making jumped_to_bootstrap volatile makes the kernel continue on with uACPI

#

where it panics

#

uACPI, TRACE: initialized GPE block _GPE[0->64], 2 AML handlers (IRQ 9)
uACPI, WARN: event initialization failed: invalid argument
uACPI Failed in uacpi_namespace_load! Status code: 7, error message: invalid argument

While initializing uACPI

#

thinkong

#

if (!IOAPIC_MapIRQToVector(irq, irqHnd->GetVector() + 0x20, true, triggerMode::LevelSensitive))
    return UACPI_STATUS_INVALID_ARGUMENT;```

#

somewhere in there

#

rather the I/O APIC is uninitialized (probably not)

#

or the redirection entry has an invalid index

#

oh wow debug info + optimizations = 💩

#

why does g++ get rid of my frame pointers

#

I told it not to

#

I think

#

-fno-omit-frame-pointer

#

I indeed do

#

whoopsie

#

my IOAPIC registers aren't volatile

#

I fixed those bugs

ashen goblet Apr 21, 2024, 10:14 PM

#

flint idol why does g++ get rid of my frame pointers

Does it still? I had the same problem a while ago

flint idol Apr 21, 2024, 10:24 PM

#

Well I never did much problem solving related to that other than guaranteeing the compiler saw -fno-omit-frame-pointer

#

I'll check to see if it's something goofy with my build system

#

It isn't

#

actually it could be

#

why doesn't the compiler command line include any optimization options

#

oh wait

#

🤦

#

I'm compiling the debug build files as verbose

thick jolt Apr 22, 2024, 9:16 AM

#

☠️

thick jolt Apr 22, 2024, 9:42 AM

#

well glad u fixed it

flint idol Apr 22, 2024, 9:56 AM

#

Oh no the problem with the frame pointer still stands

thick jolt Apr 22, 2024, 9:58 AM

#

flint idol Oh no the problem with the frame pointer still stands

damn

#

😭

flint idol Apr 22, 2024, 9:59 AM

#

Not a huge problem, only poses a problem with debugging release builds

thick jolt Apr 22, 2024, 9:59 AM

#

im sure you'll figure it out eventually omar

flint idol Apr 22, 2024, 9:59 AM

#

Eventually

thick jolt Apr 22, 2024, 9:59 AM

#

eventually

flint idol Apr 22, 2024, 9:59 AM

#

Eventually though isn't rn

#

I just woke up

thick jolt Apr 22, 2024, 10:00 AM

#

halfmemeright

#

yea

flint idol Apr 22, 2024, 11:44 AM

#

Surprisingly, the kernel only ~~uses~~ allocates dynamically 832 kib of bytes

#

According to the PMM

#

I'll add the kernel, modules, bootloader reclaimable, and the framebuffer's size to the count

#

It uses 6.25 mib including those + dynamic allocations

#

~~That's too much, time to microoptimize everything~~

real pecan Apr 22, 2024, 11:50 AM

#

why would framebuffer count lol

flint idol Apr 22, 2024, 11:50 AM

#

Because

#

uhhh

#

why not

#

I'm basing this off a nPagesAllocated count

#

It uses 2.273 mib while optimizing for size without the framebuffer counting

thick jolt Apr 22, 2024, 1:26 PM

#

meow

flint idol Apr 22, 2024, 1:26 PM

#

uhhh

#

ok

thick jolt Apr 22, 2024, 1:26 PM

#

am i not allowed to meo

#

meow*

#

i love OBOS!

#

i'd die for obos

flint idol Apr 22, 2024, 1:27 PM

#

I wouldn't

thick jolt Apr 22, 2024, 1:27 PM

#

why angr

flint idol Apr 22, 2024, 1:27 PM

#

because

#

osdev isn't life

thick jolt Apr 22, 2024, 1:28 PM

#

osdev IS my life

#

😎

#

😎 😎

ashen goblet Apr 22, 2024, 1:30 PM

#

What the fuck

thick jolt Apr 22, 2024, 1:30 PM

#

ashen goblet What the fuck

whats wrong

ashen goblet Apr 22, 2024, 1:30 PM

#

Weird af convo

thick jolt Apr 22, 2024, 1:30 PM

#

true

flint idol Apr 22, 2024, 1:31 PM

#

Please keep this thread to discussion of OBOS

#

and not meow

thick jolt Apr 22, 2024, 1:31 PM

#

flint idol Please keep this thread to discussion of OBOS

fine angr

flint idol Apr 22, 2024, 1:31 PM

#

if you want meow, ask the staff for their secret channels

#

but you didn't hear it from me

weary hound Apr 23, 2024, 8:22 PM

#

.!unmute @thick jolt

umbral iceBOT Apr 23, 2024, 8:22 PM

#

Unmuted @thick jolt

thick jolt Apr 23, 2024, 8:22 PM

#

weary hound .!unmute <@787341689333350450>

thanks qookie!

flint idol Apr 23, 2024, 8:23 PM

#

idk why in my channel but ok

thick jolt Apr 23, 2024, 8:23 PM

#

🤷‍♀️

weary hound Apr 23, 2024, 8:23 PM

#

i just happened to click here

flint idol Apr 23, 2024, 8:23 PM

#

because you happened to click here

#

leave a star halfmemeright

inland radish Apr 24, 2024, 5:07 AM

#

steep ravine keyronex is on par and im pretty sure mintia had more but it seems to me that th...

It's not gone, you can still access it

#

But its owner is banned for saying something along the lines of "imagine having your nice white party disturbed" or sth to that effect regarding the October 7 attack

#

#987861125104300083 message here

opaque pulsar Apr 24, 2024, 6:14 AM

#

Yeah

flint idol Apr 25, 2024, 9:52 PM

#

I tested OBOS on real hardware with optimizations and it works

flint idol Apr 25, 2024, 10:14 PM

#

I've compiled with the most vigorous optimization options, -Ofast and -Oz (not at the same time of course)

#

and it worked without problems

#

on qemu and real hardware

thick jolt Apr 25, 2024, 10:14 PM

#

congrats omar!

#

happy for u 😄

ashen goblet Apr 25, 2024, 10:36 PM

#

Thats great, you’re very likely bug free then

thick jolt Apr 25, 2024, 10:37 PM

#

yep!

ashen goblet Apr 25, 2024, 10:37 PM

#

First stable release of obos

thick jolt Apr 25, 2024, 10:37 PM

#

real

thick jolt Apr 26, 2024, 5:29 PM

#

obos

flint idol Apr 26, 2024, 5:29 PM

#

What about it

thick jolt Apr 26, 2024, 5:29 PM

#

i love it

#

whats the progress so far on it

limber wave Apr 26, 2024, 5:38 PM

#

thick jolt whats the progress so far on it

look at him @flint idol, he is still trying to flex

thick jolt Apr 26, 2024, 5:38 PM

#

limber wave look at him <@841762704766140486>, he is still trying to flex

im not

#

i actually wanna know his progress

#

i want this os to succeed yk

#

i like obos

flint idol Apr 26, 2024, 5:43 PM

#

thick jolt whats the progress so far on it

I'm working on VFS

#

Just still no work on it but the structs and what they do

#

I'll start tomorrow probably

thick jolt Apr 26, 2024, 5:45 PM

#

flint idol Just still no work on it but the structs and what they do

which vfs?

#

fat32?

flint idol Apr 26, 2024, 5:54 PM

#

That's not what a VFS is

#

A VFS abstracts the underlying filesystem

thick jolt Apr 26, 2024, 5:54 PM

#

ohh

flint idol Apr 27, 2024, 2:10 AM

#

Bruh

#

I get UBSan in my kernel

#

and it triple faults before initialization of the gdt

#

bruh

#

stb printf

#

Now I got a lot of UB to fix

#

(or ignore)

#

I got it to run to processor initialization

#

until it panics ofc

#

misalgined ptr

#

ignored

#

now it fails in the VMM

#

while allocating pages for the allocator

#

which was allocating for uACPI

#

Where this happens:

UBSan Violation handle pointer overflow in file C:/Code/obos/src/oboskrnl/vmm/map.cpp at line 206:9
Stack trace:
        0xffffffff8002fbe3: _ZN4obos6logger13panicVariadicEPvPKcP13__va_list_tag+308
        0xffffffff8002faaf: _ZN4obos6logger13panicVariadicEPvPKcP13__va_list_tag+0
        0xffffffff80048b6f: __ubsan_handle_type_mismatch+0
        0xffffffff80049149: __ubsan_handle_pointer_overflow_abort+0
        0xffffffff80034b76: _ZN4obos3vmmL11CanAllocateEPNS0_7ContextEPvm+288
        0xffffffff8003983b: _ZN4obos3vmm8AllocateEPNS0_7ContextEPvmmm+524
        0xffffffff8003eae4: _ZN4obos10allocators14BasicAllocator17allocateNewRegionEm+424
        0xffffffff8003ba16: _ZN4obos10allocators14BasicAllocator8AllocateEm+716
        0xffffffff8003ae3a: _Znwm+349
        0xffffffff8003afff: _Znam+24
        0xffffffff80067695: uacpi_kernel_alloc+24
        0xffffffff8004ff0a: item_array_alloc+140
        0xffffffff8005a200: exec_op+467
        0xffffffff8005b806: uacpi_execute_control_method+360
        0xffffffff80053169: do_load_table+147
        0xffffffff8005396e: uacpi_load_table+40
        0xffffffff8004dccd: uacpi_namespace_load+163
        0xffffffff800485dc: _ZN4obos5kmainEv+446```

#

Sounds like a problem for tomorrow

flint idol Apr 27, 2024, 2:42 AM

#

Y'know what UBSan is annoying (my code sucks)

#

It's somehow messing up pages that should be zeroed

#

Kind of useless too

#

I might as well implement ASAN

#

(tomorrow)

ornate ginkgo Apr 27, 2024, 5:27 AM

#

Just don't have UB for ubsan to detect 😎

ashen goblet Apr 27, 2024, 9:27 AM

#

flint idol I get UBSan in my kernel

Howd you port that

#

Do you have to provide some sort of api for it to work

ornate ginkgo Apr 27, 2024, 9:31 AM

#

it just calls some functions

#

https://github.com/rdmsr/tinyubsan

ashen goblet Apr 27, 2024, 9:33 AM

#

Oh yeah thats what it is

#

The __handles you have to provide

flint idol Apr 27, 2024, 5:29 PM

#

I think the messing up zeroed pages was somewhere else

#

because I'm getting page faults/general protection faults (depending on whether it was the first boot or not) in CanAllocate

#

It's definitely a problem in the demand page handler

#

Because that zeroes the pages on page fault

#

But

memzero((void*)newPd.virt, newPd.isHugePage ? OBOS_HUGE_PAGE_SIZE : OBOS_PAGE_SIZE);

Should keep the bytes zeroed thinkong

#

Maybe if I set them to like 0xaa

#

and see if that is the problem

#

Scheduler now hangs somewhere in init

#

Well I'll just not do that

#

and instead set memory to 0xAA in the allocator's free function

#

To fix those bugs

#

and of course I'll only do that if the kernel is being compiled as debug

#

As expected

#

An issue with non-zeroed memory while allocating

#

page descriptors/nodes

#

or even worse this is a use-after-free

#

It's even causing problems in the kernel debugger (because it calls the allocator which calls the VMM which might be using after free or not zeroing memory)

flint idol Apr 27, 2024, 5:38 PM

#

flint idol I just fixed a bug relating to the kernel not knowing the framebuffer is mapped....

Maybe it has something to do with this

#

It seems to be a use after free

#

Nope

#

Just me forgetting to zero memory

#

I think

#

My kernel seems to be cursed again

#

actually nvm

#

There is a use-after-free

#

Static analyzers can help with that I guess

#

I found nothing that useful for my problem but I guess I found a bug in gcc's static analyzer

#

      |   80 |                         if (!(--cur->referenceCount))
      |      |                         ~~                                                    
      |      |                         |
      |      |                         (2) following 'true' branch...
      |   81 |                                 delete cur;
      |      |                                 ~~~~~~~~~~                                    
      |      |                                 |      |
      |      |                                 |      (3) ...to here
      |      |                                 (4) pointer 'cur' is checked for NULL here but it was already dereferenced at (1)```

#

It thinks I'm checking if cur is nullptr

#

but I'm in fact trying to check if the reference count decremented is zero

weary hound Apr 27, 2024, 5:59 PM

#

does your operator delete contain an if (ptr)?

#

it's saying that if you take the event 2 branch, event 3/4 checks for null but event 1 (and the check for event 2) already dereferenced the pointer so the compiler assumes it is non-null

flint idol Apr 27, 2024, 6:01 PM

#

weary hound does your `operator delete` contain an `if (ptr)`?

The compiler wouldn't know what operator delete does though

#

Except assume it follows the standard

weary hound Apr 27, 2024, 6:01 PM

#

if it's inlined the analyzer could peek into it

flint idol Apr 27, 2024, 6:01 PM

#

It isn't

real pecan Apr 27, 2024, 6:01 PM

#

looks like an analyzer bug

flint idol Apr 27, 2024, 6:02 PM

#

it's in a separate definition in another file

weary hound Apr 27, 2024, 6:03 PM

#

flint idol but I'm in fact trying to check if the reference count decremented is zero

event 4 is not pointing at the reference count check?

flint idol Apr 27, 2024, 6:04 PM

#

yeah I misunderstood that

#

oops

#

use after free

#

if (!(--cur->referenceCount))
    delete cur;
vmm::Free(cur->addressSpace, (void*)cur->thread_stack.base, cur->thread_stack.size);```

real pecan Apr 27, 2024, 6:09 PM

#

oberrow and a memory bug? thats impossible

flint idol Apr 27, 2024, 6:11 PM

#

and a possible nullptr dereference somewhere else

#

in driverIdList::Remove

#

It calls Find(id)

#

then it assumes it was found

#

(it returns nullptr on failure to find the id)

thick jolt Apr 27, 2024, 6:14 PM

#

real pecan oberrow and a memory bug? thats impossible

halfmemeright

flint idol Apr 27, 2024, 6:14 PM

#

Now in driverInterface::LoadDriver I got a possible nullptr dereference

#

Rather that or it's a false-positive

#

false positive

real pecan Apr 27, 2024, 6:17 PM

#

implement asan

#

you're gonna need that shit LULW

flint idol Apr 27, 2024, 6:20 PM

#

lol

#

True though

ashen goblet Apr 27, 2024, 6:42 PM

#

real pecan implement asan

Is that as trivial as ubsan

#

I dont know where you are finding references on how to implement these

real pecan Apr 27, 2024, 6:43 PM

#

nope

flint idol Apr 27, 2024, 6:48 PM

#

After some stubbing and implementation of the __asan_load/__asan_store functions

#

I got it to link

#

But not to run

#

(Far from it)

#

I get a triple fault

#

Before initialization of the GDT

#

Bruh

#

It assumes that the framebuffer request is broken

#

(Which would be a safe assumption if it weren't for limine filling in the structure)

#

Then it triple faults because ASAN tries to break into KDBG

#

and of course no IDT

#

Holy

#

The binary is a mb with KASAN

#

on

weary hound Apr 27, 2024, 6:53 PM

#

gcc kasan replaces every load and store with a function call so

real pecan Apr 27, 2024, 6:53 PM

#

weary hound gcc kasan replaces every load and store with a function call so

Does it not on clang?

weary hound Apr 27, 2024, 6:53 PM

#

no on clang it uses the shadow space, and you have to provide an address delta to compute the shadow space address

flint idol Apr 27, 2024, 6:53 PM

#

weary hound gcc kasan replaces every load and store with a function call so

Well that makes more sense

#

Where can I find the documentation

#

Though

real pecan Apr 27, 2024, 6:53 PM

#

weary hound no on clang it uses the shadow space, and you have to provide an address delta t...

Interesting

weary hound Apr 27, 2024, 6:54 PM

#

if kasan
    # The shadow memory offset is calculated as:
    #     ShadowMemoryBase - (StartOfHigherHalf/ShadowScale).
    # This ensures that the first shadow byte of the higher half ends up at ShadowMemoryBase.
    # Lower half addresses become non-canonical by design
    # (and KASAN should not instrument lower-half access anyway).
    args += [
        '-DTHOR_KASAN',
        '-fsanitize=kernel-address',
        '-mllvm', '-asan-mapping-offset=0xdfffe00000000000',
        '-mllvm', '-asan-globals=false',
    ]
endif

flint idol Apr 27, 2024, 8:47 PM

#

flint idol It assumes that the framebuffer request is broken

nvm I just don't know how to use KASAN

#

Now I got some debugging to do

real pecan Apr 27, 2024, 8:48 PM

#

did u implement it?

flint idol Apr 27, 2024, 8:48 PM

#

Almost

#

I just need to some things with posioning or whatever it's called for a full implementation

real pecan Apr 27, 2024, 8:49 PM

#

nice

flint idol Apr 27, 2024, 8:51 PM

#

I have a LOT of things in the VMM I need to mark as "don't KASAN pls"

#

If only CMake provided a way to modify compiler options for specific files

#

oh wait it does

#

Well doing that is too much work

#

Even the smallest functions in the VMM have to have it disabled

#

(MapToHHDM)

#

After doing that

#

I get to the kdbg screen

#

Now a hang while starting processors

#

It'd be a good idea to disable KASAN in the SMP bootstrap function

#

The kernel's performance is 💩 right now

#

But I fixed the bug with my KASAN implementation that caused processor startup to hang

#

Maybe it's a bad idea to allow KASAN to run in the spin lock's lock/unlock functions

real pecan Apr 27, 2024, 9:04 PM

#

kasan on linux is like 3% overhead?

flint idol Apr 27, 2024, 9:04 PM

#

Maybe not in the allocator either

flint idol Apr 27, 2024, 9:05 PM

#

real pecan kasan on linux is like 3% overhead?

Well I can see the characters being printed one by one on the screen

real pecan Apr 27, 2024, 9:05 PM

#

lol

flint idol Apr 27, 2024, 9:06 PM

#

Solution is to stop KASAN from happening in the console's output routine

#

By the time I'm done with this, KASAN would be completely disabled everywhere lol

ashen goblet Apr 27, 2024, 9:07 PM

#

real pecan kasan on linux is like 3% overhead?

How is that possible if it hooks into every single load/store

flint idol Apr 27, 2024, 9:07 PM

#

Maybe linux's functions for KASAN are just really really fast

ashen goblet Apr 27, 2024, 9:07 PM

#

Or yours are just really really slow KEKW

flint idol Apr 27, 2024, 9:08 PM

#

well this is the core function:

    OBOS_NO_KASAN void asan_verify(uintptr_t at, size_t size, uintptr_t ip, bool rw, bool abort)
    {
        bool crossesPageBoundary = OBOS_CROSSES_PAGE_BOUNDARY(at, size);
        if (crossesPageBoundary)
            size += OBOS_PAGE_SIZE;
        vmm::page_descriptor pd{};
        bool present = false;
        size_t ps = 0;
        at -= (at % OBOS_PAGE_SIZE);
        if (size % OBOS_PAGE_SIZE)
            size += (OBOS_PAGE_SIZE - (size % OBOS_PAGE_SIZE));
        for (uintptr_t addr = at; addr <= (at + size); addr += ps)
        {
            arch::get_page_descriptor((vmm::Context*)nullptr, (void*)at, pd);
            if (!pd.present)
            {
                asan_report(addr, size, ip, rw, abort);
                return;
            }
#if OBOS_HAS_HUGE_PAGE_SUPPORT
            ps = pd.isHugePage ? OBOS_HUGE_PAGE_SIZE : OBOS_PAGE_SIZE;
#else
            ps = OBOS_PAGE_SIZE;
#endif
        }
    }```

#

Doesn't look like it'd be that big of an overhead

real pecan Apr 27, 2024, 9:09 PM

#

ashen goblet How is that possible if it hooks into every single load/store

no idea, but i think its good enough that they even enable it in release

#

like ubuntu ships with it enabled i think

ashen goblet Apr 27, 2024, 9:10 PM

#

wow

#

Seems insanely useful for debugging

flint idol Apr 27, 2024, 9:11 PM

#

Well isn't that the entire point

ashen goblet Apr 27, 2024, 9:11 PM

#

Since literally every ubuntu user actively participates in debugging

real pecan Apr 27, 2024, 9:11 PM

#

on my pc i get an out-of-bounds array access on every kernel boot

#

reported by kasan

flint idol Apr 27, 2024, 9:11 PM

#

real pecan on my pc i get an out-of-bounds array access on every kernel boot

KEKW

ashen goblet Apr 27, 2024, 9:13 PM

#

Mine fails with VFS: Unable to mount root fs to block(0,0)

real pecan Apr 27, 2024, 9:13 PM

#

looks like it can be hardware accelerated https://docs.kernel.org/dev-tools/kasan.html

#

maybe thats how they do it so fast

ashen goblet Apr 27, 2024, 9:14 PM

#

Only on arm if I read it right

flint idol Apr 27, 2024, 9:15 PM

#

Welp,

[Error] ASAN Violation at ffffffff8001e56c while trying to write 160 bytes from 0x0000000000000000.```

#

In the allocator's allocateNewRegion function

#

Oh no

#

panics are slow af

#

because they fill the background colour

#

Now that I fixed that

#

#

I got some bugs to fix

#

My kernel is cursed again

#

Function Allocate, File C:/Code/obos/src/oboskrnl/vmm/map.cpp, Line 481: Assertion failed, "!pd.present". Attempt to reallocate address ffff8000feeb68d0.
Stack trace:
        0xffffffff800140bd: _ZN4obos6logger9panicImplEPvbPKcP13__va_list_tag+237
        0xffffffff80013f6f: _ZN4obos6logger20reportKASANViolationEPKcz+0
        0xffffffff80013f42: _ZN4obos6logger13panicVariadicEPvPKcP13__va_list_tag+0
        0xffffffff80016b95: _ZN4obos3vmm8AllocateEPNS0_7ContextEPvmmm+713
        0xffffffff80018105: _ZN4obos10allocators14BasicAllocator17allocateNewRegionEm+255
        0xffffffff80017763: _ZN4obos10allocators14BasicAllocator8AllocateEm+187
        0xffffffff80017364: _Znwm+105
        0xffffffff800173f2: _Znam+24
        0xffffffff8004fa5f: uacpi_kernel_alloc+24
        0xffffffff800382d3: item_array_alloc+140
        0xffffffff800425c9: exec_op+467
        0xffffffff80043bcf: uacpi_execute_control_method+360
        0xffffffff8003b532: do_load_table+147
        0xffffffff8003bd37: uacpi_load_table+40
        0xffffffff80036096: uacpi_namespace_load+163
        0xffffffff8001b054: _ZN4obos5kmainEv+232```

#

I disable KASAN and get that

real pecan Apr 27, 2024, 9:27 PM

#

since the nyaux guy fixed his bug now its on you

flint idol Apr 27, 2024, 9:27 PM

#

nooo

#

Why hast thou put thy beshrew upon me?

#

I reboot and I get a GPF

#

Of course

#

Average debugging experience of OBOS

#

reboot again

#

PF

#

All this might as well be caused by non-zeroed memory

#

or a use after free

real pecan Apr 27, 2024, 9:30 PM

#

sounds like a race or use of uninitialized memory

flint idol Apr 27, 2024, 9:30 PM

#

Time to make KASAN more aggressive

#

actually too much work

real pecan Apr 27, 2024, 9:31 PM

#

u dont use shadow space

#

so ur kasan is not any better than a page fault handler

flint idol Apr 27, 2024, 9:31 PM

#

Shadow space I think is extra space added next to allocated blocks that shouldn't be modified

#

and if it is it's a KASAN violation right

real pecan Apr 27, 2024, 9:32 PM

#

watch a talk about how kasan is implemented in linux

#

or read an article

#OBOS (not vibecoded)