#OBOS (not vibecoded)

1 messages · Page 7 of 1

sharp pike
#

that sounds like a skill issue

flint idol
# 
      40 text files.
      40 unique files.
       5 files ignored.

github.com/AlDanial/cloc v 1.90  T=1.41 s (25.6 files/s, 2680.9 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C/C++ Header                    14            235            461           1699
Assembly                         5             94             44            596
C                                5             33             26            307
CMake                            6             33             16            148
Bourne Shell                     3              4              7             27
DOS Batch                        1              4              6             17
Markdown                         2              0              0             12
-------------------------------------------------------------------------------
SUM:                            36            403            560           2806
-------------------------------------------------------------------------------```
flint idol
#

I think if I went to the first commit with code, the build system would be larger

sharp pike
#

i mean yeah that's normal

flint idol
#

10 lines of c

#

in the first commit

#

with the gdt

#

(my gdt is in asm)

#

if you make a gdt with fancy functions and you're in long mode, reconsider

sharp pike
#

btw what is supposedly wrong about pitust's GDT

flint idol
#

idk look up

sharp pike
#

the "64-bit data" descriptor is just a data descriptor

flint idol
#

apparently data descriptors must not have the long mode bit set

sharp pike
#

correct

#

does it have it?

#

but iirc even if it has it, it's ignored

weary hound
#

and ig it's ignored if E=0

sharp pike
#

yeah

#

the sdms from both intel and amd are rather wishy washy about what is ignored and what isn't in terms of descriptor flags

weary hound
#

ig it doesn't really matter to them because why are you spending more than 5 minutes on the gdt

real pecan
sharp pike
#

i was just saying that the behaviour i have seen from pretty much every CPU is to just ignore it

#

of course, that doesn't mean that that will always be the case for every CPU released in the future

flint idol
#

Hmmm I plan for my scheduler to be able to raise thread priorities based on when they were last run

#

and if it one is being starved the priority gets raised

#

I'm thinking on how to get the time of when they were last run

#

Just I have no idea how to do with that without some sort of tick count connected to the timer irq

ashen goblet
#

Why not mix round robin and longest-time-not-run

#

Are you doing rr?

flint idol
#

well this is like a variation of longest-time-not-run

#

except if a thread is starving for CPU time, it's priority gets raised for a bit

#

iirc NT does something like this

ashen goblet
#

Yeah but messing with priorities seems risky

#

What about a seperate priority level

#

A starve priority basically

flint idol
#

the priority would basically be raised by one for two quantums, where the priority is lowered to the og priority

#

also this might also be able to allow for better load balancing

#

which I might implement

#

I can balance load by giving other CPUs threads that are starving

#

of course I don't got SMP yet

#

I should probably do that now

#

I think this might be the first hobby kernel to go GDT->IDT->Scheduler/SMP

flint idol
ashen goblet
#

Does uacpi guys bootloader enable the other cores

flint idol
#

you mean Hyper

#

it doesn't

#

it doesn't have requests like limine does

ashen goblet
#

Then youll need a bit of lapic ig

flint idol
#

The order of how I'm implementing things in this kernel is quite cursed

#

But it's fine

#

probably first hobby kernel to go 'GDT->IDT->LAPIC->SMP->Scheduler'

#

Well the LAPIC is pretty simple, I just get LAPIC address from MSR then I do magic to initialize it

#

Then LAPIC is acheived

ashen goblet
flint idol
#

idk

#

I haven't ever implemented load balancing

#

I think it's done in the scheduler somewhere

ashen goblet
#

A worker thread for that seems sensible
You give it guaranteed cpu time every 250 ms or something
Let him look at how much the cores had to work in that time
And then you move the threads around their lists

#

But you try to have a as high as possible cpu affinity

#

Because cache and tlb

#

Maybe you can even use intels performance monitoring stuff for that

flint idol
#

oops

#

we have summoned the fadanoid

ashen goblet
#

I was just joking please dont bully me i know my idea wasnt that good

flint idol
#

I don't think he's looked in this thread

#

yet

ashen goblet
#

Ping him

flint idol
#

no u

ashen goblet
#

No

#

Im scared

flint idol
#

because I can't set caching settings for the LAPIC address in the HHDM

#

and the LAPIC should be marked as UC iirc

#

or goofy stuff will happen

weary hound
#

the mtrrs should set it as UC

#

and UC in mtrr overrides any cache setting chosen by the page tables (except for WC)

flint idol
weary hound
#

having proper infra for explicitly setting caching modes for mmio is still a good idea

#

in case you want to port to aarch64 or whatever

flint idol
#

Well I'm stuck needing to get SMP without anyway to modify page table entries

#

I could just make the infrastrucutre for SMP

#

but implement it later

#

I think I might do that

#

that means load balancing is to be tested later

#

anyway, how do I mesure when a thread was last run

#

when I don't have any way to mesure time

#

obvious solution is to call time()

#

I could have an atomic counter that counts the number of scheduler calls

#

and use that

#

but that isn't very useful

#

actually it could do what I want

#

because I need that to see whether a thread is starving

#

or not

#

So it could be used because I don't necessarily need a timestamp

#

Anyway, time to add the infrastructure for SMP (cpu_local structs)

inland radish
flint idol
#

Oh and I also forgot to say I got no timer rn

#

That's what I meant by the message you replied to

#

I spent most of yesterday thinking about the scheduler

#

But today I think I'll be able to start implementation of it

flint idol
#

I want to make my scheduler lockless

#

Now I could do that, if it weren't for load balancing

#

#schedulers message

#

I could use a concurrent skiplist, but @haughty abyss says he wouldn't recommend using it except to learn how they work

haughty abyss
#

do try

flint idol
#

I haven't ever used one before, so I'll see what I can do with it

haughty abyss
#

and they are much easier to implement than rbtrees imo

flint idol
#

Is there any reason you wouldn't recommend it?

haughty abyss
#

well not difficult to implement

#

just difficult to implement (a) locklessly (b) correctly

flint idol
#

interesting

#

Well I'll deal with the scheduler later, I gtg now

near linden
#

oh wait, no memory stuff before sched?

warm oar
#

Just call NtRaiseHardError instead, no admin rights required

limber wave
warm oar
limber wave
#

I'm assuming it's a native API and it seems to be undocumented. I'll have to take a look at it

warm oar
haughty abyss
#

surely you cant just bsod as any user

warm oar
haughty abyss
#

i do not

#

also looks like you need RtlAdjustPriviledge

warm oar
#

Yes but nothing a normal user couldn't do

haughty abyss
#

fair

limber wave
#

I thought when the condrv kernel bug happened like 2 years ago, the vulnerability was that an unpriveliged user could bring down the system. TIL you could do that anyway

warm oar
#

Also ignore patchguard, it's unrelated

flint idol
#

for future reference, __thread is a keyword in GCC

#

I was getting errors related to that (I was trying to struct __thread) and couldn't figure out why until I searched __thread in GCC

#

and found out it was a keyword for TLS

nocturne swallow
#

Well, names prefixed with a double underscore are supposed to be reserved anyway

flint idol
#

fair

pearl jungle
#

also does anyone know how to scroll to the top of this

flint idol
#

I got a lot of the scheduler done

#

I'll test it tomorrow

flint idol
#

After I fixed a couple bugs with the scheduler

#

It works

#

It should be able to work both preemptively and cooperatively

#

The scheduler depends on nothing but itself

#

(not including arch-specific thread context functions)

#

and the panic function (not necessarily a dependency)

#

I wasn't 100% sure how I'd implement work stealing, more specifically when to trigger work stealing

#

So maybe there are better ways I could do that

#

From a comment in the work stealing function in the scheduler:

Compare the current list's node count to that of other cores, and if it is less than the node count of one quarter of cores, then steal some work from some of the cores.

#

Then if that condition is true, the scheduler will take some (otherCpuNodeCount-currentCpuNodeCount / 4) threads from the other CPU.

#

*try to take

flint idol
#

and then it repeats this for each priority list

#

Now that I got the scheduler though, I'll start work on the PMM

#

It should be simple enough

#

(famous last words)

#

First though I need to parse the hyper boot context

#

It'd be cool if I could parallelize kernel boot

#

But I probably cannot

thick jolt
#

do it!!!!

flint idol
#

of course I will

thick jolt
#

YES !!!!!

#

!!!!!!

flint idol
#

I've always done free list

#

except in my first two kernels

#

but they were shit

thick jolt
#

and u will only do free list because freelist is the best

#

!!!

thick jolt
#

bitmap?

flint idol
#

indeed

thick jolt
#

ew

flint idol
#

but before that, I need to commit the scheduler code.

thick jolt
#

hyper boot v limine who would win

flint idol
#

Hyper is faster

#

tbf I had the timeout to three seconds on limine

#

Hyper does not have any sort of boot menu from what I've seen

#

Hyper is less beginner-friendly

#

For example, it assumes the kernel will remap itself with proper permissions

#

It maps the kernel as RW

#

also no magic sections in the boot protocol Hyper uses

thick jolt
flint idol
#

apparently with modules Hyper is a lot faster

#

I'll see limine on my previous kernel vs hyper on my current kernel

#

(I'll disable the timeout on limine)

thick jolt
#

i use a 3mb module to load a wallpaper for flanterm, it still loads that pretty fast

thick jolt
vernal chasm
#

Freelist + bitmap + lookup table of range sizes
Speedy PMM :>

flint idol
#

Too much work for something so simple

vernal chasm
#

Dude it's like 300 loc

#

I think lmao

flint idol
#

I want to spend 10 minutes on PMM

#

then forget about it

vernal chasm
#

Been a while since I wrote it

vernal chasm
flint idol
#

No thank you

vernal chasm
#

oof

flint idol
#

what is it even licensed under

vernal chasm
#

MIT

flint idol
#

Good

#

do you agree that GPLv3 license is evil

#

and shouldn't exist

#

Imagine if Linux was MIT KEKW

vernal chasm
vernal chasm
#
  • 19 for the "magic" functions
uint64_t FreeLUTGetValue(uint8_t index)
{
    if (index < 192)
        return (uint64_t) index + 1;
    return (1UL << (index - 192)) + 192;
}

uint8_t FreeLUTGetFloorIndex(uint64_t value)
{
    if (value < 193)
        return (uint8_t) value - 1;
    return (254 - __builtin_clzll(value - 192));
}

uint8_t FreeLUTGetCeilIndex(uint64_t value)
{
    if (value < 193)
        return (uint8_t) value - 1;
    return (255 - __builtin_clzll(value - 193));
}
flint idol
#

Anyway scheduler works enough

#

time to merge PR

inland radish
flint idol
#

I made the PMM, and now I have to test it

thick jolt
#

dwarf seems complicated

flint idol
#

Maybe use libdwarf then

thick jolt
#

dont think thats freestanding

flint idol
#

Libdwarf has been focused for years on both providing access to DWARF2 through DWARF5 data in a portable way

thick jolt
#

oh yeah it can

#

do u use it in OBOS?

flint idol
#

No

#

ofc not

#

I only have a scheduler rn

#

and an untested PMM

thick jolt
#

ah okay

flint idol
#

If and when I make a kernel debugger, I'm probably using libdwarf

thick jolt
#

i need one rn for my kernel cause debugging real hardware is a pain

flint idol
#

you can get symbols for the address from the symbol table

#

I might implement UBSan and KASAN after the PMM

#

Actually KASAN would be useless without dynamic memory allocation

#

But UBSan will be helpful

thick jolt
real pecan
#

That's clearly testing stuff

thick jolt
#

okay ill delete that directory

flint idol
#

The pmm seems like it works

#

Including with alignment values (0x200)

#

I'll try allocating more and more pages

#

until it shits itself

#

I'll allocate 2mib first

#

that works

#

how about 2 gib (shouldn't work)

flint idol
#

Well there was a bug

#

When trying to allocate something but no memory could be found for it

#

I fixed that though

#

Now I can allocate physical memory

#

So what shall I do next

vernal chasm
#

How do you allocate now?

flint idol
#

Allocate what?

#

Dynamic memory?

vernal chasm
#

With the PMM

#

Rather how does it do it?

flint idol
#

freelist

vernal chasm
#

Yes but like how do you do it

#

As in how do you find the correct range of pages to allocate

flint idol
vernal chasm
#

Since a plain freelist wouldn't work all that well with sized allocations...

flint idol
#

a node contains a count with the amount of pages it represents

#

it does goofy things with the page count based on the node and alignment value

#

then if the page count <= the node's page count

#

it allocates part of that node

vernal chasm
flint idol
#

Any better ways to do it

#

?

#

by chance?

#

(using different alloc strategy doesn't count)

vernal chasm
#

I mean you could always keep track of a basic lookup table for known sizes and instantly jump to the first node of that known size

flint idol
#

I was talking about the alignment thing

vernal chasm
#

Oh I just check if ((node.base + alignmentMask) & ~alignmentMask) + size <= node.base + node.size

#

Where alignmentMask = alignment - 1 generally...

#

iirc

flint idol
#

Well I'm making alignment an arbitrary value passed by the user of the function

vernal chasm
#

Dont

#

Keep it power of 2

#

If someone wants 3 page alignment, give them hell :>

flint idol
#

good idea

#

I made it so page alignment has to be a power of two

#

Merged PMM

#

I implemented UBSan but kept it out of the PMM branch

#

I'll just push to master

#

I pushed UBSan to master

#

I got UBSan

#

But I got no way to allocate virtual memory

#

(except by allocating physical pages then offsetting the address to the HHDM)

#

Obviously that is not very convinient

vernal chasm
#

Good luck :>

flint idol
#

I'll get page table management

#

then I can implement VMM

#

How I'll implement the VMM, idk yet.

real pecan
#

is it stable rn

flint idol
#

I think so

#

It doesn't trigger UBSan

#

it doesn't page fault in goofy ways

real pecan
#

damn

#

This might be the last rewrite

flint idol
#

before I give up /j

#

I'll get a way to manage allocate/unmap virtual memory

#

then I'll make an allocator

#

Then I'll make an abstract IRQ interface

#

Then I'll make the scheduler preemptive

#

Then I'll make the VMM

#

Then VFS

#

Then driver interface

#

should keep me busy for quite a bit

#

actually switch VFS with VMM

#

VFS->VMM

#

then driver interface

#

I might implement some stuff in between (kernel debugger/KASAN)

vernal chasm
#

What I've always wondered tho, what do you do if there's no page table on some architecture?

#

But instead they do some other shit for virtual memory

#

just always assume page table?

real pecan
flint idol
#

unless you consider segmentation virtual memory

vernal chasm
real pecan
#

it will be dead on arrival

vernal chasm
#

Are you sure?

real pecan
#

because no one is going to adapt the code for them

#

yes

vernal chasm
#

What if that magical new method was 50% more efficient?

real pecan
#

also like

vernal chasm
#

Or even more...

flint idol
#

Well I only plan on supporting semi-sane architectures (x86-64, aarch64)

real pecan
#

virtual memory is already abstract enough

#

on x86 its technically not even called virtual

#

its linear

#

virtual is segmentation

vernal chasm
#

The x86 page-translation mechanism (or simply paging mechanism) enables system software to create separate address spaces for each process or application. These address spaces are known as virtual-address spaces. System software uses the paging mechanism to selectively map individual pages of physical memory into the virtual-address space using a set of hierarchical address-translation tables known collectively as page tables.
Literally the first part of chapter 5 in amd doc 2

real pecan
#

yeah but the address is called linear

vernal chasm
#

I mean to the process the addresses seem linear and normal, but for the cpu it's virtual

real pecan
#

on x86 its
linear -> page table transform -> virtual -> segmentation transform -> physical

vernal chasm
#

Cool ig

#

But from what I've read in the amd doc they refer to it as virtual address, not linear address

real pecan
#

maybe its an amd thing

devout niche
vernal chasm
#

I don't really read the intel doc

#

Because it's just too overly complex

#

xD

devout niche
flint idol
#

what I can do

#

Isn't that what windows does

devout niche
#

they say only "logical address" and "linear address"

image.png
devout niche
flint idol
#

what about keyronex

devout niche
#

also keyronex

vernal chasm
flint idol
#

Most of the next stuff I plan on implementing are relatively simple

#

page allocation (not VMM) stuff, allocator (I'm porting my previous kernel's one), and an IRQ interface

#

and making the scheduler preemptive

#

That should take a week or two if I actually work on it

#

Page allocation/freeing will be done with void* OBOS_AllocatePages(size_t) and void* OBOS_FreePages(void*, size_t)

#

They will be used solely in kernel-mode

#

ie. there won't be any matching syscalls

#

User-mode will be using the VMM

real pecan
#

Where OBOS_MaybeSegfault

flint idol
#

I know what to do

#

Add no exception handlers

#

so that when I page fault, it hangs

#

why?

#

because

#

of course I probably won't do that

#

Allocation of pages will be done as followed:
Search nodes for free region
Map free region
Add node for free region (nodes will be stored intrusively)
Return region address

#

Freeing will be done as followed:
Unlink region node
Unmap region

sharp pike
flint idol
#

I meant magic numbers in variables

#

I'm pretty sure limine finds requests by searching the kernel binary for some UUID

#

or something

sharp pike
#

yes, but that's not what you said lol

#

it doesn't use any ELF segment and/or section and/or symbol or anything else

#

it's actually ELF agnostic

flint idol
#

Maybe it'd be cool if I could have limine and hyper support in the kernel

#

because Hyper doesn't have a boot menu from what I've seen

real pecan
sharp pike
#

Limine is a boot manager and bootloader

#

ig Hyper is only the latter

#

which is fine

real pecan
#

Yup

sharp pike
#

not every bootloader needs to be a boot manager

flint idol
#

I got OBOS_BasicMMAllocatePages working

#

Which means I can port my old kernel's allocator

#

But before that I want to quickly implement OBOS_BasicMMFreePages

vernal chasm
flint idol
#

Allocates pages

vernal chasm
#

In what way xD

#

Same as the pmm?

flint idol
#

It's for virtual memory

real pecan
vernal chasm
#

I could tell, but like does it just use a freelist?

vernal chasm
flint idol
#

It keeps a list of allocated regions

vernal chasm
#

"How to allocate memory, Step 1) Restart computer, Step 2) Steal memory"

flint idol
#

and looks for free memory in between

#

I got OBOS_BasicMMFreePages working I think

#

I'll test this by allocating more regions, and trying to free them

#

and if that works I'm porting the allocator

steep ravine
flint idol
#

I just need to test this a bit more

#

bug when trying to allocate more than once

#

it will map the same region twice

#

I think I know the problem, so now I'll apply the fix

real pecan
#

me when oberrow says it works after testing the single non-edge-case scenario and that didnt crash

flint idol
#

I just said it works

#

I had barely tested it

real pecan
#

oberrow the fortune teller

flint idol
#

At least I'm testing now and not "random shit go!"

real pecan
#

yea

flint idol
#

There is like no edge case I can think of with these functions, except "I tried to allocate the entire kernel address space." But hopefully no one will do that

#

I'll try to think

#

about any edge cases

#

and test them

#

because as we all know, I am very good at creating memory-safe code /s

real pecan
flint idol
# 
void* mem1 = OBOS_BasicMMAllocatePages(4*0x1000 , nullptr);
void* mem3 = OBOS_BasicMMAllocatePages(32*0x1000, nullptr);
OBOS_BasicMMFreePages(mem3, 0x20000);
void* mem2 = OBOS_BasicMMAllocatePages(16*0x1000, nullptr);
OBOS_BasicMMFreePages(mem1, 0x4000);
OBOS_BasicMMFreePages(mem2, 0x10000);```
I used this to test the functions
#

It seems to work

#

i.e., everything gets mapped and unmapped

#

no memory gets corrupted

#

nothing shits itself

#

I'll try adding another allocation

#

so I can further try to fragment it

# 
void* mem1 = OBOS_BasicMMAllocatePages(4*0x1000 , nullptr);
void* mem3 = OBOS_BasicMMAllocatePages(32*0x1000, nullptr);
OBOS_BasicMMFreePages(mem3, 0x20000);
void* mem4 = OBOS_BasicMMAllocatePages(48*0x1000, nullptr);
OBOS_BasicMMFreePages(mem1, 0x4000);
void* mem2 = OBOS_BasicMMAllocatePages(16*0x1000, nullptr);
OBOS_BasicMMFreePages(mem2, 0x10000);
OBOS_BasicMMFreePages(mem4, 0x30000);
vernal chasm
#

Now use randomness :>

flint idol
#

Too much work

vernal chasm
#

You could just use the hardware rand instructions, tho they are bad...

flint idol
#

too much work

vernal chasm
#

bruh

steep ravine
# 
uint64_t seed;
asm volatile("rdseed %0" : "=r" (seed));
#

hard

flint idol
#

no

steep ravine
#

much work

flint idol
#

I'm just saying I don't want to

steep ravine
#

no, you said too much work

flint idol
#

yeah

#

because I'm too lazy

empty kernel
flint idol
#

This would be for a test case

#

So that's not really needed if I know it will be supported

#

Anyway, the new test case worked

#

I've pushed that code

#

now I gotta port the old kernel's allocator

flint idol
#

Before I port the old kernel's allocator, I have to setup my own page tables

#

I spent the past half-hour writing code for mapping the kernel

#

It seems to map it correctly

#

So now I just have to map the HHDM

#

Then I can port the allocator

flint idol
#

Now that I make my own page tables

#

I can port the old kernel's allocator

#

I also implemented huge pages, because otherwise, mapping the HHDM would take way too long.

vernal chasm
flint idol
#

Maybe...

#

But first I have to fix a bug with mapping the kernel

#

I fixed it

#

It's time to port the old kernel's allocator

#

then port uACPI to use as a test

#

after porting the allocator

#

I'll implement KASAN

vernal chasm
#

Yk I was thinking of a kinda ok thing for nullptr testing, what if you reserve the lowest and highest 1 GiB pages thonk

flint idol
#

then you can't use them

#

Anyway it's not like I access nullptr in my code anywhere (not)

#

I'll see what the static analyzer says

flint idol
#

But I beleive it's a false positive

vernal chasm
flint idol
#

I've ported the allocator

#

Time to see if it works

#

(it probably doesn't)

#

UBSan Violation type mismatch
thinkong

#

When I blk->magic = PAGEBLOCK_MAGIC;

#

I fixed it

#

Well I can allocate 32 bytes

#

Time to put it up some allocator test

#

and see if it shits itself

#

(it probably will)

flint idol
#

It indeed does..

#

in OBOS_BasicMMAllocatePages

#

I think I fixed that bug

#

The allocator passes the tests

#

except rdrand always gives 0 (which gets rounded up to 16), so the allocator isn't really getting tested

flint idol
#

After fixing the problem with my RNG

#

(bad inline asm)

#

The allocator still works

#

I'll give it 1 million passes and a max allocation size of 10 pages

#

see what happens

#

10000 passes takes a lot longer

# 
  _____ _______ ____  _____
 / ____|__   __/ __ \|  __ \
| (___    | | | |  | | |__) |
 \___ \   | | | |  | |  ___/
 ____) |  | | | |__| | |
|_____/   |_|  \____/|_|
Kernel Panic! Reason: 7. Information on the crash is below:
No more avaliable physical memory!```
#

I also changed my panic screen

#

I'm going to make a makeshift text renderer

#

and test this on real hardware

ashen goblet
#

Reason: 7

flint idol
#

OBOS_PANIC_NO_MEMORY

#

I should probably translate that to something readable

vernal chasm
#

7

flint idol
#

Ok I got the text renderer ported

#

time to test on real hardware

vernal chasm
flint idol
#

Well I ported it

#

so I can get logs on real hardware

#

obvious solution is to make my own motherboard so I can get port 0xe9 logs

#

It hangs sometime before initialization of the framebuffer...

#

rather that or it's triple faulting really really fast

vernal chasm
#

oof

flint idol
#

I'll parse the boot context earilier

#

so I can get the framebuffer eariler

#

Well it's hanging on initialization of the scheduler

#

I will debug that

#

It was uninitialized memory

ashen goblet
#

every C programmers daily nightmare

#

i had that same problem a few times

#

i tried to find some qemu option to force it to randomize its memroy

#

but couldn't find anything

#

that'd be very nice if they added smth like that

flint idol
#

Ok it seems like the allocator tests work (enough) on real hardware

#

It's pretty slow because of all the memory being allocated

steep ravine
flint idol
#

Time to make the IRQ interface, but before I do that, I'm adding support for the LAPIC, IOAPIC, and SMP

#

after the IRQ interface, I'm making the scheduler preemptive (that will be fun to debug)

#

I just pushed the code for the allocator

#

Oh yeah I was going to implement KASAN

#

and in this case, by implement I meant Copy+Paste the old kernel's implementation (- the cpu mmu emulation, as 8infy calls it)

flint idol
#

I've implement KASAN

#

so I think I'll merge the allocator branch

real pecan
flint idol
#

uhhh

#

both

real pecan
#

Bruh

flint idol
#

it'll be fine

#

Well what it does, is it first checks if the memory is mapped with proper flags

#

then it checks if the memory is poisoned

flint idol
flint idol
#

no way I have 2000 additions for a branch I started yesterday

#

(tbf I did copy a lot of code from the previous kernel)

#

I want to optimize KASAN

#

Simple solution: Remove MMU Emulation

#

damn that's already a lot faster

#

visibly

#

the allocator is really slow with KASAN enabled

#

I think I should just disable KASAN for the allocator

#

It goes through many many passes with allocations of random sizes and very little frees

#

without faulting

#

it only gets slower, but that's expected because there's a lot of regions

#

that the allocator needs to go through

#

It might be a good optimization to move regions with no free nodes into a separate list

#

Maybe I could use some different data structure that allows for faster search times

#

I've heard skip lists are good for this

#

Except I'm too dumb to understand skip lists

#

In about a week, I already have a lot more than my previous kernel did in the first week

#

actually nvm

#

I have around the same

#

I've removed MMU emulation from KASAN

#

I got an allocator

#

and KASAN

#

Time to merge PR

flint idol
#

I just love it when the optimizer just breaks what seems to be perfectly fine C code

#

how is it causing asm functions to stop working properly

#

I just love it when I zero memory

#

then gdb shows the memory as not-zeroed

#

nvm gdb is just broken with optimizations

#

Oh wow

#

The bug is because I accidentally used rax instead of rdi

weary hound
#

in where?

flint idol
#

CoreS_SetupThreadContext

#

It was checking if the first parameter was nullptr

#

the first parameter is a pointer to where the context is

#

and I accidentally used rax instead of rdi while comparing

weary hound
#

ah

flint idol
#

so it would exit prematurely in this one specific case with optimizations enabled

#

where I'm assuming rax was nullptr

#

otherwise, there are no problems with optimizations

#

So tomorrow I'll probably start implementing the LAPIC, the I/O APIC, and SMP

#

shouldn't take too long considering I can Copy+Paste a lot of code from my previous kernel

#

For some reason though I enjoy making SMP Trampolines

#

With SMP comes making the scheduler preemptive and supporting TLB shootdowns

#

well I don't need to make the scheduler preemptive because of SMP

#

but it'd make things a lot easier if I do

#

anyway that's enough osdev for today

vernal chasm
flint idol
#

I did for the previous kernel

#

but I will

flint idol
#

wait how were messages I sent unread thonk

#

discord told me they were unread

empty kernel
#

discord likes to do that sometimes

thick jolt
#

funny discord

flint idol
#

I got a proper stack guard implemented (it uses rdrand/rdseed at the beginning of the kernel to figure out the stack guard value)

#

Except if neither of those instructions are supported

#

then it just uses its default value

#

It also puts a bogus return address on the stack before jumping into the kernel entry in the off-chance that the kernel entry decides to return to initiate a triple fault

#

Because the stack guard is quite light in terms of implementation, I'll leave it on by default

#

I'll just push it

#

because why not

#

I accidentally made a merge commit while pulling

#

because I forgot to pull my changes on test subject 2

#

oh whoopsie

#

bug

#

if the carry flag is clear, then I should restart because a random number wasn't generated

#

should be as simple as jnc to go back to the top

#

it'd be funny if I make the installer for the os be a qr code scanner

#

the qr code is like a tar image

#

that contains the other essential parts of the system (like drivers and the init system)

#

the kernel will just be copied from the current kernel binary

#

ig

#

this would be funny, except qr codes can store a maximum of ~3kib of data

flint idol
vernal chasm
#

compression

#

Do the multi disc install method, but with multiple qr codes

flint idol
#

I barely have a kernel

#

I'll think about installer later

#

if anything it's going to be a glorified dd command

#

If I ever do decide to do it

thick jolt
#

Omar how’s progress on obos

flint idol
#

uhh

#

got stack protector

#

which is cool

#

when are you going to get UBSan/KASAN/Stack Protector

thick jolt
#

For my kernel?

flint idol
#

who else's

thick jolt
#

Uhhhh good question actually

#

maybe tomorrow

flint idol
#

UBSan is easy

#

Stack protector is easier

#

(literally one function and some global variable)

#

KASAN can be made simple

#

using poison values

#

only problem is I can't find documentation

thick jolt
#

Sorry poison values?

flint idol
#

there is a shadow space

#

before or after an allocated memory block

#

that is filled with some value

thick jolt
flint idol
#

It's some place of memory that isn't to be touched

#

basically what the compiler does with KASAN on is it calls a function (with an access size parameter or the access size is in the function signature)

#

which would compare the bytes at the memory address with some given "poison" values

#

for example, say 0xfd was used by your KASAN implementation for freed blocks

#

and some thing tries to access a freed block

#

KASAN would catch that memory access and report use-after-free

#

or if you fill the shadow space after a memory block with 0xfa

#

you could detect out-of-bounds accesses on the heap

#

the stack smasher is pretty simple, at the beginning a function the compiler pushes some value on the stack

#

then at the end, before returning it verifies the value popped is the same that was pushed

#

I have no idea how UBSan works

#

also there's threadsanitizer

#

for detecting data races, etc.

#

I might implement that as well

thick jolt
#

That’s a lot to implement

flint idol
#

in a nutshell it's:
memset

#

then

#

memcmp

#

then if memcmp returns the same

#

you panic

#

well not actually

#

I'd say read my implementation

#

but don't

#

Starting out with stack protector is easiest

#

It's one function

#

and one variable (set to a random value)

#

you CANNOT mess up on it

#

unless you are pretty darn stupid

thick jolt
#

Then I will implement that tomorrow

flint idol
#

my entire implementation of stack protector is like 10 lines

#

basically you define this function: __stack_chk_fail

#

which is called

#

if the stack protector finds a violation

#

you should probably panic if that happens

#

then you define a variable (of word size) called __stack_chk_guard

#

which contains some random value

#

then you let compiler do magic

thick jolt
#

that it?

flint idol
#

for stack protector, yes

thick jolt
#

tf

#

that’s easy

flint idol
#

it's pretty simple because it's implemented in a simple way, push value in __stack_chk_guard onto stack when I'm called, then before returning, check if the value we popped is the same as the value in __stack_chk_guard

#

now my kernel tries to randomize the value for the stack protector value at runtime

thick jolt
#

Might fuck up my interrupt stubs tho

flint idol
#

your interrupt stubs aren't in C are they

thick jolt
#

No

flint idol
#

then how would it

thick jolt
#

ok fair enough

flint idol
#

also it's not on for every function

thick jolt
#

What time is it for u Omar rn

flint idol
flint idol
#

nor is it late

#

not the time to sleep

#

yet it is after sunset

thick jolt
#

What’s your time zone again

flint idol
#

UTC-20

thick jolt
#

my dads gonna kill me if I don’t sleep

flint idol
#

bruh

#

osdev isn't more important than sleep

#

I say you sleep

#

your dad spares

#

you

#

and you might get to do osdev tomorrow

thick jolt
#

okay

#

go

#

gn*

flint idol
#

gn

lean glen
flint idol
#

I just noticed I don't lock my allocator

#

whoopsie

ashen goblet
#

How did it survive that long 🤣

#

Oh this is obos 5.0 so you probably didn’t have smp/multithreading yet

flint idol
#

After taking a small break

#

I've finally added a lock to the allocator

thick jolt
#

easy

#

😎

flint idol
#

Time to get to APIC

#

and IRQ interface

#

and I/O APIC

#

and SMP

#

and preemptive scheduler

vernal chasm
# flint idol I've finally added a lock to the allocator

I have been thinking of a method of making a fast multi threadable PMM, so essentially you split the entire memory range into equally sized regions (say like 8 different regions), then when you go to allocate you do an atomic fetch add with 1 to some index, and use that index to allocate within one of those memory regions, in that way you'd only have to lock and unlock a smaller region of memory.

Something like

std::atomic_uint32_t g_Index = 0;

struct Allocator
{
  std::atomic_bool Lock = false;
  // Additional stuff to make Allocator bigger...
};

Allocator g_Allocators[8];

void* Allocate(size_t size, size_t alignment)
{
  do
  {
    uint32_t index     = g_Index.fetch_add(1) & 7;
    auto&    allocator = g_Allocators[index];
    {
      // Lock
      bool expected = false;
      while (expected = false && !allocator.Lock.compare_exchange_strong(expected, true))
        ;
    }

    // Do the allocation, if there's not enough space in the region you'd need to try again in a new region.

    // Unlock
    allocator.Lock = false;

    if (found)
      return ptr;
  } while (true);
}
ornate ginkgo
#

my kernel wasnt that complex at the time, but I remember noticing that only the first two regions were used a lot of the time, so I'd be curious to know how beneficial this could be.

vernal chasm
#

Yeah idk xD

#

Haven't tried this method

flint idol
#

I got LAPIC

#

working

#

time for SMP shenanigans

thick jolt
#

funny

flint idol
#

hmm should I parse the MADT

thick jolt
#

when userspace

flint idol
#

or should I just send an IPI to all cpus

flint idol
thick jolt
#

bru

vernal chasm
flint idol
#

I ain't rushing to user space

#

that isn't cool

vernal chasm
#

Use the broadcast wake up

thick jolt
#

userspace doesnt take months

flint idol
#

your vmm is shit

#

that's why it took you 23 days

thick jolt
#

bru

#

😭

#

thats a little mean

flint idol
#

bruh

#

you can't even allocate pages with specific prot flags

thick jolt
#

and

#

why does it matter

flint idol
#

security

thick jolt
flint idol
#

no XD bit

thick jolt
#

i do not give a shit about security

flint idol
#

no secure

flint idol
thick jolt
#

whos gonna use this as a

flint idol
#

still

thick jolt
#

heckin

#

yk

flint idol
#

I will trl

thick jolt
flint idol
#

jk

#

anyway

flint idol
vernal chasm
flint idol
#

do you mean INIT and startup IPIs

vernal chasm
#

I.e. the broadcast one

#

Yes

flint idol
#

with the destination shorthand set to all but self?

vernal chasm
#

Yes

flint idol
#

I've heard that their can be defective cores

#

that could be started up by that

vernal chasm
#

They shouldn't

flint idol
#

ok

#

time for my weirdest SMP trampoline

vernal chasm
#

If they do then the entire cpu is faulty

flint idol
#

lemme brainstorm how I'll do this

empty kernel
flint idol
#

good point

#

MADT parsing here I go

empty kernel
#

globally sending fixed or NMI IPIs is fine though.

vernal chasm
#

Because defective cores should be "fused" off

empty kernel
#

you would hope the manufacturer would do that, but there is always a possibility that they haven't

flint idol
#

never forget

#

OEMs are fucked in the head

empty kernel
#

exactly

vernal chasm
flint idol
#

I'm targeting at least x86_64-v2

vernal chasm
flint idol
#

I hate having to find my SDT/MADT structs

#

and put them in a new file

#

every time I rewrite

#

while I'm doing that I might as well copy over the HPET table as well

real pecan
flint idol
#

too late

real pecan
#

Unlucky

vernal chasm
flint idol
#

I just finished MADT parsing code

#

It's dead simple

#

find MADT in XSDT

#

find LAPIC ids in MADT

vernal chasm
flint idol
#

I feel like, despite testing the allocator this much

#

it will still fail while allocating the cpu info

vernal chasm
#

weakling

sharp pike
vernal chasm
#

That probaly hurt lmao

flint idol
#

My scheduler and SMP are doin some funky stuff

#

something is setting a newly initialized cpu's current thread

#

to some idle thread

#

that hadn't been created yet

#

(and yet is still completely valid by the looks of it)

#

wtf

#

What in the actual fuck is happening

#

My kernel is predicting the future

#

There are readied idle threads for each uninitialized cpu

#

my brain hurts

#

HOW

#

Note: This state is being observed before any SMP trampoline is executed

#

the scheduler

#

state

#

is correct

#

This is literally all that's done to the cpu state:

cpu_local* cpu_info = (cpu_local*)OBOS_KernelAllocator->Allocate(OBOS_KernelAllocator, s_nLAPICIDs*sizeof(cpu_local), nullptr);```
#
  • a memcpy for the bsp cpu state
#

The scheduler only gets called once, on the first yield

#

maybe it was unzeroed memory

#

a clean boot shows zeroed memory

#

for the cpu info

#

Currently SMP works for one cpu

#

any more triple faults

#

hmm well it's not the extra cpus

#

it's on execution of ffffffff8001e120, Arch_IdleTask

#

which seems to be mapped XD

vernal chasm
flint idol
#

probably being put in the data section

vernal chasm
#

Since the british law specifies that anyone who predicts the future is a witch :>

flint idol
#

nah it was just non zeroed memory

vernal chasm
#

Skill issue

flint idol
#

and since my kernel is so predictable

#

it happened to be the right values

flint idol
#

so the kernel was allocating the same memory as a previous reboot

#

without zeroing it

#

Anyway, SMP works

#

all CPUs but the BSP are on the idle task

#

the BSP is halted

#

because the kernel is done booting

#

time to push changes

#

@thick jolt when SMP in nyaux

thick jolt
#

listen im fucking going through pain that is my vfs

#

please no

flint idol
#

It's only 500 lines

thick jolt
#

please no

vernal chasm
#

Meanwhilst I'm trying to figure out how to get an accurate timer frequency from another timer O_O

flint idol
#

NMI broadcasts cause a triple fault

#

oh wait I'm dumb

#

I never initialize the IDT on the APs

#

Tomorrow I'll probably work on getting an IRQ interface

#

then I'll probably work on timers

#

then I'll make the scheduler preemptive

flint idol
lean glen
thick jolt
flint idol
thick jolt
lean glen
lean glen
#

then smp will be easy

flint idol
# lean glen oh

for example, if you wrote some memory at physical address 0x10000 (assuming that's available in the memory map), then rebooted, then read the memory at 0x10000, it should return the same value that you had written before

vernal chasm
ashen goblet
flint idol
#

I'm back in oberrow hell

#

this time with memory corruption in the allocator

#

It's a problem with my makeshift virtual memory allocator

flint idol
#

A region in the allocator is getting corrupted

light jacinth
#

oberrow's curse always returns

flint idol
#

maybe

#

if I make KASAN guard against writes to allocator structures

#

if I just add some sort of shadow space there

#

too much work

#

I'll just look at the diff between now and HEAD really carefully

#

hmm

# 
for (uintptr_t addr = base; addr < (base + sz); addr += OBOS_PAGE_SIZE)
{
    uintptr_t phys = 0;
    OBOSS_GetPagePhysicalAddress((void*)addr, &phys);
    if (phys)
        OBOSS_FreePhysicalPages(phys, 1);
    OBOSS_UnmapPage((void*)addr);
}```
One of these lines is the culprit
#

what if I reorder the unmap and the free of the physical page

#

It's a bug with Arch_FreePhysicalPages

flint idol
#

In trying to fix that bug

#

I end up making the kernel unmap the stack

flint idol
#

It seems to be a bug with my function for getting a physical address from a virtual address halfmemeright

#

The statement for getting a physical address from a PT of one level above it in all its glory:
Arch_MaskPhysicalAddressFromEntry(((uintptr_t*)Arch_MapToHHDM(Arch_MaskPhysicalAddressFromEntry(entry)))[AddressToIndex(at, (uint8_t)isHugePage)]);

#

ok I fixed the bug

#

Pushed the fixes

#

Time for the irq interface

#

too boring

flint idol
#

ok so how it will work is:

Some driver wants IRQ for some reason.
It can request a vector at a specific IRQL, or it can choose the vector itself (will probably be used for stuff like ACPI)
To implement irq sharing, each driver that wants to register an IRQ has got to register a callback that returns a boolean stating whether thats their irq or not. On IRQ, the irq interface will iterate each irq object for the IRQ's vector, and call each callback until a driver reports the IRQ was theirs, and on that case, the irq interface will call the handler for that driver, then return from the IRQ.
On the case that the driver disabled work sharing for its irq vector, the irq interface simply calls the handler of the IRQ.
In the case that a driver asks for work sharing to be disabled on initialization of one of its irq objects, and makes the 'force' parameter true, the irq interface moves all irq objects on that vector to another vector of the same IRQL. That is, unless, one of the irq objects chose its vector, in that case, the operation fails. Otherwise, the operation is successful.
If force is false, but the vector has no irq objects registered on it, the operation succeeds.
The IDs of IRQ Vectors are arch-specific, and the platform decides the type of the ID. The arch must have (or simulate) at least one ID per vector for each valid IRQL (2-15, any IRQLs <= 1 cannot be used for initializing an IRQ object.
An IRQ object must have a handler.
#

Now time for the boring-er part

#

implementing this nooo

#

tl;dr: This IRQ interface provides a way for the kernel+drivers to get their IRQs handled in an arch-independent way, abstracting away irq sharing

#

The IRQ dispatcher will send EOIs automatically

#

It will get an interrupt frame it can pass on to any handlers

thick jolt
#

expect with my vmm this time

flint idol
#

No I got out pretty quickly

flint idol
#

I need to implement a hashmap for the irq interface nooo

#

It's trivial, but annoying

inland radish
#

Lets think about the problem

#

You need a data structure where you can get elements by index

#

And the indices are relatively small (you only have 256 IDT slots if you're thinking of MSI and the like)

#

It can't hurt to just have an array of linked lists

real pecan
#

A hash map for 256 entries is insane

#

Maybe if it was 256 thousand

haughty abyss
#

at some point reallocating the hashmap becomes quite difficult

#

like, resizing a 200k entry hashmap means you need to allocate 400k entries worth of hashmap space which may be "a bit" tricky

real pecan
haughty abyss
real pecan
#

At that point an array is better because locality

haughty abyss
#

crucially, its also really good if your entries are noncontinous

#

like lets say your keys are strings, or random numbers in 0..2^64

devout niche
#

why would you need a hashmap for interrupt handlers though?

#

vectors aren't going to be shared that much

#

a linked list ought to be enough for anyone in this case

flint idol
flint idol
#

I've implemented a lot of the IRQ interface

#

I just need the irq dispatching thingy

#

I'm pretty sure that's all I need at least

flint idol
flint idol
#

I've tested it

#

Time to implement IRQ dispatching ig

flint idol
#

and I made sure the irq vectors and irq objects don't have bs values

#

The IRQ dispatcher is relatively simple, it sends an EOI, then if irq sharing is disabled, call the irq object's handler and return from the IRQ, otherwise loop over the IRQ objects until one reports that this IRQ is theirs, then call that irq object's handler and return

flint idol
#

IRQL is being very goofy rn

#

cr8 is zero

#

but the irql reported by the cpu local variable is 15

#

gs_base is cleared thinkong

#

It can't have anything to do with goofy scheduler preemption, as the scheduler doesn't have preemption yet

#

The stack is getting corrupted

#

it seems

#

the rbp at the time of the IRQ is shit

#

rbp+8 does not have the return RIP

#

oh well the IRQ happened on execution of push rbp

#

so nvm

#

wtf happened

devout niche
#

i was going to ask if you f no omitted frame pointer or whatever the flag is

#

but that's moot now

flint idol
#

so far, it seems as if gs_base got reset by magical forces

devout niche
#

is there potential for an interrupt to arrive between ISR entry and swapgs, or between swapgs and iret?

flint idol
#

unless I do and forgot to account for it (likely, I copied the ISR stub from my previous kernel, which copied its ISR stub from the kernel before it)

#

it does swapgs

devout niche
#

if you aren't ever entering into userland it isn't likely to be a real problem, but if you are, you are in trouble

flint idol
#

so if I just write to KERNEL_GS_BASE it should solve my problem

devout niche
#

statistically it's unlikely that if you jamp to junk memory that you'd have a wrgsbase among it, even less likely wrmsr

#

so either your ISR isn't robust or some jinn is playing with your CPU

flint idol
#

well KERNEL_GS_BASE was zero

#

on the swapgs in the ISR stub

#

so that seems to have been my problem

#

I'll test it

devout niche
flint idol
#

why

devout niche
#

that is not customary

chilly osprey
#

yeah that can cause issues

devout niche
#

it is customary to swapgs on leaving userland and then again on returning

#

otherwise, you might as well not swapgs at all

flint idol
#

So should I just swapgs if an interrupt happened in user land

#

otherwise don't

chilly osprey
#

well you'll need to for syscalls at some point

devout niche
flint idol
# 
    cmp [rsp+0xB0], 0x8 ; Kernel code
    je .no_swapgs1
    swapgs
.no_swapgs1:```
#

then I repeat that before the return from the ISR stub

devout niche
#

i do the opposite but that's fine too

testl $3, 16(%rsp)
devout niche
chilly osprey
flint idol
#

Well they're disabled until the IRQ dispatcher gets called, so I'm fine there

chilly osprey
#

is using nasm so bad

flint idol
#

oh wait nvm

#

I got my clis and stis backward

devout niche
# chilly osprey my eyes

does nasm do 68k? but i like the way GAS gives similar syntax for amd64 and AT&T:

    move.l %a1@(44), %sp@ | load new saved-%pc
flint idol
#

wtf am I reading

#

anyway

chilly osprey
#

there is a small issue here which is the unlikely event that you receive another interrupt before reaching the swapgs
in which cause you would end up being in the kernel, receiving an interrupt from the kernel and thus not swapping gs, and now you are in the kernel with userland gs

devout niche
chilly osprey
flint idol
#

no no

#

I sti on entry

#

well not exactly

chilly osprey
flint idol
#

the IRQ dispatcher calls CoreS_EnterIRQHandler on entry of the function

#

which stis

chilly osprey
#

right