#The Flopperating System (tfos) - x86 32 bit Hobby OS

1 messages · Page 3 of 1

crimson plover
#

: )

bold grove
crimson plover
#

rip

#

dont let it run too far

bold grove
#

I’ll send you a paste bin in a bit

#

I have that assembly function

#

I’m just

#

Idk

#

Bro :(

#

Now it’s just stuck crashing

#

Like in a loop

#

Not even —no-shutdown does anything

#

and it keeps pointing to garbage in my kernel just random points

crimson plover
#

lmk when u can vc lol

bold grove
#

Yeah shouldn’t be too long

bold grove
#

okay well i at least got that crashing issue fixed

#

@crimson plover look at this

#
thread_t* thread_create(int (*fn)(void*), void* arg, uint32_t* stack) {
    if (!stack) {
        stack = kmalloc(STACK_SIZE);
        stack = stack + (STACK_SIZE / sizeof(uint32_t));
    }

    thread_t* t = thread_alloc();

    *--stack = 0x200;
    *--stack = 0;
    *--stack = 0;
    *--stack = 0;
    *--stack = 0;
    *--stack = (uint32_t)arg;
    *--stack = (uint32_t)fn;
    *--stack = (uint32_t)_thread_entry;

    t->esp = (uint32_t)stack;

    thread_enqueue(t);
    return t;
}
crimson plover
#

idk what to make of it

rigid root
#

you should really check those allocations for null

bold grove
crimson plover
#

ngl I just panic inside of my pmm if I run out of mem

#

lol

#

future me will hate me

primal nymph
#

btw @bold grove i had some similar crash, it's better if you do that in asm

#

the problem was setting the stack value in c++

#

this is how i ended up doing it

bold grove
primal nymph
#

i was setting the next entry of a linked list to itself

#

and it took me 2 days to find why

bold grove
#

@primal nymph this is the issue I was having for over a month

#

Bexause I’m just

primal nymph
#

lol

#

btw, how does a thread know when it should be stopped?

#

like, do you have to manually set it to terminated state?

primal nymph
#

ah f

bold grove
#

Wait no

#

thread_kill() does that itself

primal nymph
#

syscall?

bold grove
#

Nah not for now but I’ll make it one

primal nymph
#

this is how i do it

#

i could easily not even set it to TERMINATED

#

bc i just call schedule again

bold grove
#

Well

#

I can get into the initialized state of the scheduler but

#

Idk

#

I’ll send a paste bin later

#

Maybe you could
Help me out

primal nymph
#

k

bold grove
#

Damn boys

#

I haven’t

#

Updated this in a while

#

I’ve been working on the sch*^#ler

#

Schedulers are cancer. Not really but , you know

#

My actual algorithm is fine

#

I use a doubly linked list

#

For thread queues

#

And I can queue threads

#

But

#

Context switching is just fucky

primal nymph
#

@bold grove

#

and if you need help send the code

bold grove
#

yeah im looking sorry

#

i will soon

#

take alook

primal nymph
#

wait, i see you have vscode open

#

i can join you if you want

bold grove
#

I’m like pretty close to get Rid of the issue

primal nymph
#

gg

bold grove
#

Bro

#

It was bexause

#

I was calling a c function

#

Which fucked up the value of esp

primal nymph
#

XD

#

how

bold grove
#

From assembly

primal nymph
#

i do that too

#

oh wait no

#

i removed that bc it gave me problems too

bold grove
#

it was _thread_confirm_esp (uint32_t *esp)

#

And when I passed eax

primal nymph
#

lol

bold grove
#

It like

#

Fucked up somefhing

#

Giving me a bullshit ESP

#

and eventually my instruction pointer was literally nowehere

#

The instruction pointer was 00FF0077 then a bunch of weird characters

#

Qemu info registers was

#

Just

#

00000000

primal nymph
#

XD

#

well, at least you figured it out

bold grove
#

Yeah

#

Now

#

I can call _thread_entry

#

And yield back to sched

#

But

#

Once I switch tasks like

#

Eventually the esp gets fucked yo again

#

Like the offset is fucked up like

#

I’m pushing the correct offset then I check it in C and it’s wrong

primal nymph
#

are you saving it correctly tho?

bold grove
#

Yes

#

I followed literally

#

The intel manual

#

And I’m saving it correctly

#

I think it’s coming down to an issue with my allocator which is funny

#

Okay not my allocator but

#

The slab is just

#

Being weird

primal nymph
#

why slab

bold grove
#

I lowkey might just

bold grove
primal nymph
#

slab might be one of the worst ones for threads

bold grove
#

Yeah I kinda like realized that

#

Now I’m just doing

#

Vmm_malloc

primal nymph
#

are you also sharing kernel pages?

bold grove
primal nymph
#

you should

bold grove
#

I’m making a region for threads

#

Is that bad

primal nymph
#

yeah prob

#

i made my allocator using contexts

bold grove
#

Okay I’ll just map to kernel pages instead

primal nymph
#

so every process gets it's own context

bold grove
#

Yeah

primal nymph
#

process, not thread btw

#

doing one per thread might be bad

bold grove
#

Multiple threads can make up a process and they share the same address space

bold grove
#

I gotta update this damn repo fuck

#

Sorry guys I’ve just been working on memorizing this stupid page of tchaikovsky

bold grove
primal nymph
#

pain

bold grove
#

It’s what I’m doing for my senior violin recital

primal nymph
#

lol, there is italian in there XD

#

that is so complicated tho

#

i only ever seen the simple ones XD

bold grove
#

It’s all Italian

#

Nah it’s pretty bad

bold grove
#

I would send a recording but I’m a butt ugly Bosnian so no thank you

primal nymph
#

lol, dw

noble remnant
#

memorising music sucks

bold grove
#

Not even 3 in

bold grove
#

average git commit

bold grove
#

So the scheduler is done

bold grove
bold grove
#

yessss

bold grove
#

well

#

now that this scheduler is done

#

i think i can restrcuture the source code

#

i might just name it

#

something else

#

idk

pliant oriole
#

ayyy good job

bold grove
#

okay im fixing some stuff

#

in the scheduler

#

some issies

#

arose

#

the function pointer inside of my thread_t was being pushed with the wrong offset

#

and i couldnt figure out where

#

so

#

but im fixing it

bold grove
#

fucking hell

bold grove
#

Aight

#

We are good

#

Time to restructure the source code

#

It’s gonna be

#

arch/
drivers/
fs/
mm/
sched/
userspace/
kernel/
flanterm/
multiboot/
limine/

#

Something actually reasonably

#

So imma have ia32

#

Which is what I’m doing now

#

But I’ll do arch separation

#

So I’ll get that limine c template

#

and make it use that for booting

#

So I can get long mode

#

And limine supports multiboot protocol so

#

Yeah imma do that tonight okay ty for reading :)

small hound
bold grove
#

next commit hopefully yes

small hound
bold grove
small hound
#

how will you balance and sync it globally

bold grove
small hound
#

that doesnt mean anything

bold grove
#

oh wait

#

my bad

small hound
#

you only have one global queue of tasks how does each core know what to schedule lol

bold grove
#

no im gonna implement smp

small hound
#

wat

#

ok i think u might not be getting what I mean lol

#

so you have some tasks you wanna run

#

and you need to figure out how to assign each core to a given set of tasks

#

how will you do that

#

i'm gonna make core-local queues with periodic global resyncs and rebalancing

#

so i have a global circular doubly linked list and i take portions of that list for each task

bold grove
#

oh wait cant i do priority scheduling

small hound
#

💥

#

wdym

bold grove
#

wait

#

cant i have each cpu core check for the highest priority scheduled task like

small hound
#

what does this mean

bold grove
#

in like a round robin

small hound
#

why would this change anything tho

#

you cant run the same task on multiple cores at the same time

bold grove
#

i know

small hound
#

💥

#

so why does it matter who is running the high priority tasks

#

just give them longer timeslices

#

💥

#

if (task_is_high_priority && timeslice_isnt_up) dont_schedule_next();

#

💥

bold grove
#

i see what you mean

torpid viper
bold grove
#

hi guysss

#

im redoing the thread structsss

#
#include <stdint.h>
#include <stddef.h>
#include <stdbool.h>
#include "sched.h"
#include "thread.h"
#include "../mem/alloc.h"
#include "../apps/echo.h"
#include "../mem/pmm.h"
#include "../mem/vmm.h"
#include "../lib/logging.h"
#include "../lib/str.h"
#include "../lib/assert.h"
#include "../mem/utils.h"


typedef void (*thread_fn) (void *arg);
typedef enum {
    THREAD_READY,
    THREAD_RUNNING,
    THREAD_EXITED,
    THREAD_DEAD
} thread_state_t;

typedef struct thread_list_node {
    struct thread_list_node* next;
    struct list_node* prev;
} thread_list_node_t;

typedef struct {
    thread_list_node_t* head;
    thread_list_node_t* tail;
    size_t count;
    thread_list_node_t node;
} thread_list_t;

typedef enum {
    USER,
    KERNEL
} thread_mode_t;

typedef struct thread {
    thread_fn fn;
    uint32_t thread_id;
    thread_state_t state;
    thread_list_node_t node;
    thread_mode_t mode;
    uintptr_t esp;
    uintptr_t base_esp;
} thread_t;

what do you think
and im just having threads point to C functions
which im assuming is fine
because i can enter whatever from a c function
once i get elf working

small hound
#

still no x64 💥

#

lgtm

bold grove
bold grove
#

Yeah

#

Fine

#

Just a different struct

small hound
#

when are we getting x64 support and testing on real hw ⁉️

bold grove
#

Runs on real hw

#

X64 support will come when it does I’m not sure

small hound
#

what does "run" mean

#

does that mean it booted faulted and gave up

bold grove
small hound
#

oh wowa

#

what about drivers

bold grove
#

Yup. keyboard works just fine

#

I can acpi power off and on

small hound
bold grove
#

Yeah but most computers have ps2 emulation

small hound
#

i wonder how lenient qemu actually is with device drivers

small hound
#

all 3 of my disk drivers might suck on real hw if disks love to fail

#

i have no fail state checks

bold grove
#

The only one that would transfer well is the NVME

small hound
#

what about ahci and ide

bold grove
#

Because it’s specification is solid and it’s not unreliable

small hound
bold grove
small hound
bold grove
#

On my old kernel

small hound
bold grove
bold grove
small hound
#

so does that mean i need to re-attempt the command like a floppy if it fails

small hound
bold grove
small hound
#

was the disk in question highly problematic

bold grove
#

512 Samsung evo ssd

#

Very new

#

And I know it works

#

I think the issue

#

Is that most hobbyist drivers

#

Don’t really account for

#

Cancer

#

That happens on real hardware

small hound
#

are you sure that these fancy new sata drives work fine with ide emulation

bold grove
#

Well it was one of these

#

So

small hound
#

i dont think these new guys play well with IDE controller support

bold grove
#

Oh well

small hound
#

they would probably like ahci better (perhaps)

bold grove
#

Yeah never got around to that

#

Idk tho

#

Because I was able to

#

Like

#

Read from sectors

small hound
#

ahci easy peasy you should get to it

bold grove
#

And detect them fine

bold grove
#

And we will finally have

#

64 bit

#

finally not just ia32

small hound
#

what was the logic behind just doing ia32 at first

#

are you rolling ur own bootloader

bold grove
#

Nah multiboot 1

bold grove
small hound
#

is that not your own bootloader

small hound
bold grove
#

Multiboot is

small hound
bold grove
#

A gnu boot specification

small hound
bold grove
small hound
#

interesting lore

bold grove
#

Yeah it’s also why I did only vga text mode for a long time and vga graphics mode

#

Because my dad made games for his own OS

small hound
#

😲

#

awesome sauce

bold grove
#

And he made this super sick spreadsheet application in vga text mode that is lowkey still really good

small hound
#

most sane 90's c programer

bold grove
#

C programmers were on one 20 years ago

timber terrace
#

i wish my dad did shit like this he just made hard drives spin to sound like the doom soundtrack

bold grove
#

My dad now does embedded systems at General Electric

small hound
#

does he make their engines too

bold grove
#

No he makes the software that allows them to work

small hound
#

when will we see an ada kernel developed here

#

has there bee one

small hound
bold grove
#

Honestly like

#

I’m very close to a new commit with a better structure

small hound
#

huge commits with 1 line messages

bold grove
#

Absolutely

#

So

#

This next commit will come tonight but

#

The main deal right now is that I want to make syscalls

#

After I tidy up my filesystem

small hound
#

do you think my kernel is okayish in organization (shameless plug)

bold grove
small hound
#

neat

#

i was making things up while organizing it

bold grove
#

Sam

#

Same*

small hound
#

menix is well organized (the legacy branch)

bold grove
#

It’s aight

#

Managarms source code hurts my head

small hound
#

it is strangely organized

#

me when i kernel/thor/generic/thor-internal/arch-generic/cursor.hpp

bold grove
#

Its structure is strange but

#

Mainly like

#

Lemme just show an example

small hound
#

u get used to it

bold grove
#

dead os

#

sad days

#

im workig on some stuff though

#

let me cook

#

maybe

sleek mulch
#

We wait

bold grove
#

Don’t worry guys im just depressed

#

Work on the os hasn’t started again but

#

It will soon I hope

small hound
bold grove
#

Although it wasn’t my goal

#

I’ll probably follow the posix api

#

For that

#

I plan to make all the software for it though so

#

Maybe I can do my own thing

small hound
#

i thought this was not a unix-like os

bold grove
#

Yeah

#

It isn’t

#

So forget what I said lol

#

I’m just gonna cope

#

Currently

#

I can

small hound
#

what 😭

bold grove
#

Sorry im bad at typing

#

Give me a bit and I’ll explain what I’m up to

bold grove
# small hound what 😭

void thread_enqueue(thread_t* thr)
Works and now I can create processes. Many threads can be attached to a process and share the same address space. I’m using a round robin and im going to implement SMP eventually

small hound
#

neat

bold grove
#

My first process is going to be the reaper

#

Then im gonna implement syscalls

#

And then make an actual shell

#

The worst part right now is

#

fork() FOR FUCKS SAKE

small hound
#

on-disk filesystems

#

block device drivers

#

usb

#

networking

#

audio cards

bold grove
#

You’re right

#

I already got a lot done

#

I’ll push later

#

And then I’ll make a proper VFS

#

and then I’ll implement NVME

small hound
#

why not do those now not later

crimson plover
#

I like working on core stuff just not worrying about getting to userspace or anything

#

hell I am like 200 todo items away from starting to port anything

bold grove
#

When I get the urge to leave this bed after 45 hours I’ll do it

small hound
bold grove
#

I’ve ran out of like weed and coffee so

#

I’m kinda just on pause rn

bold grove
bold grove
#

Todo: not make dogshit OS

#

okay

#

So

#

Working on VFS rn

small hound
bold grove
#

I’ve gotten basic operations

#

And proper vfs nodes and stuff

#

Now I can mount my temporary file system to /tmp

#

Now

#

I’m thinking

#

Either fat or ext2

#

I’ve implemented EXT2 in userspace before

#

So

#

Maybe I should do that but FAT isn’t that hard from what I’ve heard

#

My old ext2 thing for userspace was like 2000loc from 2018

#

So

#

Time to die looking through that

small hound
bold grove
small hound
small hound
bold grove
#

I made a JVM like environment

small hound
#

truncate, allocate, free, mkdir, rmdir, some basic integrity checks, all of it?

small hound
bold grove
small hound
#

I have 2 different walk functions for blocks and dirents

bold grove
#

Remember. Making it in userspace requires less low level stuff

#

So

small hound
bold grove
#

When you delete a file

#

It zeros

small hound
bold grove
#

All of its child blocks

small hound
#

neat

bold grove
#

Yes. I’ll post it sometime

#

Or

small hound
#

you probably shouldn't do that unless secure delete is specified tho

#

since like

#

slow

bold grove
small hound
#

but cool that it works

bold grove
#

—secure-del

small hound
bold grove
#

Yes

small hound
#

but like a file or large stream

bold grove
#

Exactly

small hound
#

oh that makes life easier

bold grove
#

Yeah so

#

It was simpler than implementing it for an operating system

small hound
#

very cool that you got it all to work

bold grove
#

It was painful

small hound
#

did you also fsck to see if fsck is happy about your impl

#

that was a little annoying because there were linux specific fields in the superblock for me

#

I just ignored em

#

fsck can yap all it wants

bold grove
#

Yeah no I didn’t do the Linux field

#

Fuck that shit

small hound
#

can u send code soon

#

me curious to see how this is done in userspace

bold grove
#

I mean honestly

#

Just check out

#

@west scarab

#

Look at his

small hound
#

no but where is yours

bold grove
#

Not posted because it’s ass and I hate the way it looks

small hound
#

it's not that bad i won't hate lol

#

I wrote some diabolical XHCI stuff earlier

#

would be hypocritical

bold grove
#

It will be posted in about a week or so. The only issue is that I leak some memory when removing directories

#

Idk where

small hound
#

dang wow we got a shipping delay on an internet repository

bold grove
#

Yes sir

small hound
#

😢

#

how come

bold grove
#

Because I just don’t like it right now and it’s scattered into a bullshit api

#

And written in c++ which is its own issue

#

And it won’t even compile now after gcc updates

small hound
#

what did bro do

#

compiler behavior hacks is crazy

#

idc if it compiles lol I'm just curious to see how these are done purely in userspace

bold grove
#

Well I’ll upload it soon but for now look at mishakovs

#

I’m also just not at my old computer so

#

And im too fucked up rn to go up the stairs

#

Also my girlfriend is clingy to me rn lol

#

She isn’t letting me get up bc she’s cuddling me

#

Sleepy girlfriend = terrifying

#

@small hound also for ext2 it’s extremely unnecessary to implement the Linux specific fields

#

It doesn’t affect posix compatibility

#

However , if you want to use Linux software eventually you might need to but idk

bold grove
#

Like in an os dev context those are useless

small hound
#

i never implemented them

bold grove
small hound
#

did you have sanity checks in your filesystem implementation

small hound
bold grove
small hound
#

how would i verify filesystem integrity after modification

small hound
bold grove
#

Sanity checks

small hound
#

do you validate an inode's fields as being "sane enough" for the operation to continue

#

give an example

#

show like a code snippet

#

or something

bold grove
#

Bet I don’t have it but I’ll just type it up real quick

small hound
#

no you can just say it

#

tell me like what kind of checks you had in place before operating on a possibly corrupted inode

#

cos i will probably add this soon

bold grove
#
if (inode->size > (inode->blocks * BLOCK_SIZE)) {
        return false; 
    }
#

Throw some sort of error

#

Instead of false

#

And then

#

This is useful

#
if (inode->nlinks == 0 || inode->nlinks > MAX_LINKS) {
        return false;
    }
small hound
#

oh intersting ok

bold grove
#

Also

bold grove
#
if (inode->blocks > 0) {
        bool all_zero = true;
        for (int i = 0; i < 15; i++) {
            if (inode->block[i] != 0) {
                all_zero = false;
                break;
            }
        }
        if (all_zero) {
           // give error maybe idk 
        }
    }
small hound
#

what huh

small hound
#

huhhuhuh

bold grove
#

Yes

small hound
#

no no i'm saying

#

this is not corruption

#

you can have an inode with no blocks allocated to it

#

why would you produce an error here

#

💥

bold grove
#

No no, you would check this BEFORE writing to an inode’s block for the first time

small hound
small hound
#

an file that you just created with no data would have no blocks assigned to it

#

this is throwing an error for a perfectly valid scenario

#

crazy lol

bold grove
#

There was some point where I checked something like that but yet again

small hound
#

perfectly valid filesystem throwing an FS_CORRUPT would be wild

small hound
#

💥

bold grove
#

I explained this wrong

#

before assigning blocks to an inode for the first time, do a sanity check to make sure the inode does not already reference any blocks

#

If the inode has NOT been referenced

#

This doesn’t apply to like device nodes or empty files

small hound
#

just zero the blocks

#

if you have a lazily deleted inode that just removes the entry, of course it would reference blocks

bold grove
#

Well because if that was happening im the first place something somewhere else had to have been fucked up

small hound
#

no?

#

you dont have to fully delete/zero all the contents of an inode when you remove its link

#

in ext2 you only have to remove it from the bitmap and remove all the child links from the bitmap (deallocate them)

#

so on-disk there will still be blocks on that inode when you re-allocate it

#

but it's fine, normal, and expected behavior

#

💥 did bro actually write this driver

bold grove
bold grove
small hound
#

you wrote an ext2 driver at age 12?

#

wow

small hound
#

yea i'll give u some slack bro wasn't even a teenager

#

🔥

bold grove
#

Apparently I wasn’t even 12

#

Because it’s dated to feb12 2019 but my birthday isn’t until March 29

small hound
#

ok now i'm starting to doubt you wrote this at all

bold grove
#

I did, at that point I had already made a bootloader and snake game on gameboy in assembly

#

I’d been into low level stuff for a while

#

And stuff like that

small hound
#

age 11 ext2 driver?? wowzas

bold grove
#

And my dad got me a book about file systems

#

And the C programming language

small hound
#

and yet bro is still crashing out about schedulers 6 years later

#

reasonable

bold grove
#

Ofc

#

I’m not crashing out over schedulers atm tho

#

Because it actually is fine now

small hound
#

when will we drop the 11yo amar ext2 code 🙏

bold grove
#

Fork() works

bold grove
small hound
#

i might add support for my OS to boot on it for no reason

bold grove
#

It’s not necessarily a boot loader

small hound
#

why did u call it a bootloader

bold grove
#

The game boy basically boots into a point of memory

#

Well

#

Because technically it is one

#

But the gameboy doesn’t have the facilities for me to be able to call it a true bootloader

small hound
bold grove
#

Yeah I do

#

My girl is asleep so lemme get my old laptop rq

small hound
#

awesome sauces

#

what's with ur internet harassment bot you made 😭

#

is this an april fools joke

bold grove
#

Don’t worry about it

#

Plus insta

#

Nuked it

#

So It doesn’t work

small hound
#

no your functions dont exist

#

lmao crazy stuff

bold grove
#

Okay here’s the bootloader

#

This syntax is so cancer btw

#

Have fun reading any of that to be honest

small hound
#

neat

bold grove
#

The game boy register set is

#

A B C D E H L

#

Also combined

#

Pairs

#

16 bit ones

#

AF
BC
DE
HL

small hound
bold grove
#

It’s based off of it but yeah

small hound
#

they look identical lol

#

typical reverse engineering experience

bold grove
#

I didn’t actually use anything online like

#

I actually got the rom itself

#

And objdump the fuck out of it

small hound
#

and you.... changed symbol names and called it ur own ?

#

like these are bar-for-bar identical lol

bold grove
#

Because it’s the only way you can boot anything on a gameboy it has to be that way

#

I’ll show you the guide I followed

small hound
#

wow the instructions are all identical lol

bold grove
#

It has to be

#

256 bytes

#

In that exact order

#

No exceptions

small hound
#

so this wasnt much of a bootloader then

#

just objdump -> change symbols -> recompile

#

darn

#

did u ever do any other niche hardware

bold grove
#

I own an Apple 2

#

And I made many basic programs for it

#

That’s the extent

small hound
#

neat

#

very nice pc

#

that's all pretty cool 💥

#

lucky kid lol

bold grove
#

Let me see

#

What I can do for that

#

Fuck a VFS

small hound
#

💥

bold grove
#

Now I’m purposely gonna add Easter eggs

#

Okay I figured out how to actually flow the code differently

small hound
#

'flow the code'

bold grove
#

Alright guys

#

Enough bullshit

#

Enough stupid source structure

#

Rewriting

#

Using limine

#

I already got Flanterm and pmm working :)

#

And gdt idt

#

I’m gonna rename it Zarugan (Za - roo - gun) which is the name of my fictional language I made

torpid viper
bold grove
#

long mode

#

im making a build system

#

for shits and giggles

blissful grove
bold grove
#

Wanna look at my new build system? Just started making it

#

Here’s the syntax, I already made the ast and parser and stuff so

#
decl cc = gcc;
decl src csource= main.c; // src is a built in
decl bin dest= build/main; // bin is a built in
decl args cargs = -foo;

method all (main.c) {
   @cc @cargs -o @dest;
}
small hound
#

"We have make at home"

bold grove
#

Idk it compiles my kernel quicker

#

And also compiles qemu decently quick with some problem spots lol

small hound
#

are you making another 50,000 line commit called "fixes."

bold grove
#

In 64 bit

#

Using limine

#

And I have some stuff

small hound
#

"fixes."

+50000
-0

crimson plover
#

lol

bold grove
#

“Fuck this shit. Fixes”

+10000000
-0

bold grove
#

Dead os

#

I’m not as good at this shit as I need to be

small hound
#

🥀💔

bold grove
#

I rewrote paging.c 🙏

small hound
#

is this os still dead

small hound
bold grove
#

I’m actually working on it a lot rn

bold grove
small hound
#

oooooooooooooo

bold grove
#

Nah nah nah I just rewrote my vmm and paging shit

#

And now cleaning up scheduler

#

I made all threads kernel threads

#

It’s just user mode threads are just kernel threads that jump to my user mode entry routine in C

blissful grove
#

Nice, good to see you're still going with this

bold grove
#

I’m trying lol

bold grove
#

I’ve rewritten many parts

#

Of mem/ and sched/

bold grove
#

void ctx_switch(cpu_ctx_t* _old, cpu_ctx_t* _new) {
    asm volatile (
        "pushf\n"
        "pusha\n"
        "movl %%esp, %[old]\n"
        "movl %[new], %%esp\n"
        "popa\n"
        "popf\n"
        "ret\n"
        : [old] "=m"(*_old)
        : [new] "m"(*_new)
        : "memory"
    );
}
small hound
#

why did you not call this the flopperating system

bold grove
#

I’m gonna do that

pliant oriole
small hound
#

did u fix it

#

😃

pliant oriole
#

sorta

#

somehow mov rsp, 4096 right before i jump to userspace fixed it

small hound
pliant oriole
#

nah but i just finished cr3 switching so now the scheduler is basically done and i'm happy to do a rewrite

bold grove
#

virtual memory manager done

#

the rewrite

bold grove
#

@small hound i wrote a goated vmm

small hound
#

oh ok

#

cool beans

#

I thought there would be a bit more when you said it was goated

bold grove
#

im making it a bit better

#

addibng comments and also todos

#

and some other fun methods

small hound
#

generic architecture agnostic paging system when

#

use a for loop to map pages and have each architecture implement an operation to do one layer of paging

small hound
#

also doing a full tlb flush after any paging action is kinda wild

#

the invlpg in question:

#

why does your region unmapping also deallocate the pages in your physical memory allocator 🤨

#
int unmap_range(uintptr_t vaddr, size_t size) {
    size_t pages = _align_up(size) / PAGE_SIZE;

    vm_region_t *cur = vm_regions, *prev = NULL;

    // iterate through the region list to find pages to unmap
    while (cur) {
        // check if the current region overlaps with the range to be unmapped
        if (cur->start == vaddr && cur->end == vaddr + size) {
            // if the current region is the first in the list, update the head of the list
            if (prev) {
                prev->next = cur->next;
            }
            // if the current region is the last in the list, update the tail of the list
            else {
                vm_regions = cur->next;
            }
            pmm_free_page(cur);
            break;
        }
        prev = cur;
    }
    // iterate through the pages to be unmapped
    for (size_t i = 0; i < pages; i++) {
        if (unmap_page(vaddr + i * PAGE_SIZE) < 0)
            return -1;
    }
    return 0;
}
``` sus
#

dunno why you'd wanna deallocate pages in page unmapping logic

bold grove
#

im pretty sure i pressed enter too many times in copilot so im ficing that now

#

@small hound this should be better

#

int unmap_range(uintptr_t vaddr, size_t size) {
    size_t pages = _align_up(size) / PAGE_SIZE;

    vm_region_t *cur = vm_regions, *prev = NULL;

    while (cur) {
        // check if the range is within the current region
        if (cur->start == vaddr && cur->end == vaddr + size) {

            if (prev) {
                prev->next = cur->next;

            } else {
                vm_regions = cur->next;
            }
            pmm_free_page(cur);
            break;
        }

        // move to next region
        prev = cur;
        cur = cur->next;
    }

    for (size_t i = 0; i < pages; i++) {
        uintptr_t va = vaddr + i * PAGE_SIZE;
        if (unmap_page(va) < 0) {
            // todo: handle error
            return -1;
        }
    }
    return 0;
}
bold grove
#

here:

#
void _flush_tlb(void) {
    uintptr_t cr3;
    __asm__ volatile("mov %%cr3, %0" : "=r"(cr3));
    __asm__ volatile("mov %0, %%cr3" :: "r"(cr3));
}
small hound
bold grove
small hound
bold grove
#

oh

#

isnt taht

small hound
#

on x86 this would be via an IPI

bold grove
#

yeah

small hound
#

do you have that

bold grove
#

no but

#

i had in my old src file

#

gonna go get it

#

wait

#

isnt it literally

#

just

#

asm volatile ("invlpg (%0)" : : "a" (vaddr));

#

in my case

small hound
#

yes and you need a way to send that address to other processors and wait for them to all invalidate that entry in their TLBs

#

flushing the entire TLB after changing one page mapping is very inefficient

bold grove
#

how would i be able to sen that addr to other processors

small hound
#
static void do_tlb_shootdown(uint64_t addr) {
    uint64_t this_core = get_sch_core_id();
    uint64_t cores = scheduler_get_core_count();
    for (uint64_t i = 0; i < cores; i++) {
        if (i == this_core)
            continue;

        struct core *target = global_cores[i];
        atomic_store_explicit(&target->tlb_shootdown_page, addr,
                              memory_order_release);
        lapic_send_ipi(i, TLB_SHOOTDOWN_ID);
        while (atomic_load_explicit(&target->tlb_shootdown_page,
                                    memory_order_acquire) != 0)
            cpu_relax();
    }
}``` I cheesed this a bit with an atomic ptr
bold grove
#

so if i dont have many cores rn

#

can i just

#

asm volatile ("invlpg (%0)" : : "a" (va));

#

for now

small hound
bold grove
#

and i dont have lapic just pit

#

yeah eventually but

#

not at the moment

small hound
#

ok then just remember to do the tlb shootdown

bold grove
#

okay

#

but is asm volatile ("invlpg (%0)" : : "a" (va)); good enough for now after unmapping?

small hound
#

yep 👍

#

do you support 2MB pages

#

not sure if 32 bit supports that

#

pretty sure it does

bold grove
#

yes

#

its not that off from 64'

#

just remember 4gb limited mem

#

besides that its the same shit

small hound
#

also where is your actual page table walking logic

#
int map_page(uintptr_t vaddr, uintptr_t paddr, PageAttributes attrs) {
    vaddr = _align_down(vaddr);
    paddr = _align_down(paddr);

    // get/create page table
    PTE* pt = _get_or_make_pt(vaddr);
    if (!pt) return -1;

    // map physical frame to virtual address
    uint32_t pt_idx = _pt_index(vaddr);
    attrs.frame_addr = paddr >> 12;
    SET_PF(&pt[pt_idx], attrs);

    _flush_tlb();
    return 0;
}```
#

where are you walking the page tables

#
void vmm_map_page(uintptr_t virt, uintptr_t phys, uint64_t flags) {
    if (virt == 0) {
        k_panic("CANNOT MAP PAGE 0x0!!!\n");
    }

    bool interrupts = spin_lock(&vmm_lock);
    struct page_table *current_table = kernel_pml4;

    for (uint64_t i = 0; i < 3; i++) {
        uint64_t level = virt >> (39 - (i * 9)) & 0x1FF;
        pte_t *entry = &current_table->entries[level];
        if (!ENTRY_PRESENT(*entry))
            pte_init(entry, 0);

        current_table =
            (struct page_table *) ((*entry & PAGING_PHYS_MASK) + hhdm_offset);
    }

    uint64_t L1 = (virt >> 12) & 0x1FF;
    pte_t *entry = &current_table->entries[L1];
    *entry = (phys & PAGING_PHYS_MASK) | flags | PAGING_PRESENT;

    invlpg(virt);
    do_tlb_shootdown(virt);
    spin_unlock(&vmm_lock, interrupts);
}``` like I have this here
#

the loop

#

but where do you walk your page tables

bold grove
#

ohhh i see

#

i hadthat implmented '

#

in paging.c

#

but forgot to add that subroutine

#

in map page

#

so let me

#

fix that

small hound
#

💀 this is like the most important part of your virtual memory management

#

is this blob of 300 lines of code you sent tested at all

#

does this even run

bold grove
#

i literally wrote it all today

#

and yes it runs just having issues but that was the prob lem

small hound
#

and copilot too

bold grove
#

im tired

#

nah

#

copilot i just use for cmments sometimes

small hound
bold grove
#

it initializes fine

#

but yeah

#

that was the issue

small hound
#

how is it initializing if your mapping logic doesn't walk the page tables...

#

💥

#

if vmm_init(void) calls _map_kernel(pstart, ksize), and

_map_kernel(uintptr_t pstart, size_t size)
calls map_page(vaddr, paddr, k_pg_attrs), and
map_page(uintptr_t vaddr, uintptr_t paddr, PageAttributes attrs) does this

int map_page(uintptr_t vaddr, uintptr_t paddr, PageAttributes attrs) {
    vaddr = _align_down(vaddr);
    paddr = _align_down(paddr);

    // get/create page table
    PTE* pt = _get_or_make_pt(vaddr);
    if (!pt) return -1;

    // map physical frame to virtual address
    uint32_t pt_idx = _pt_index(vaddr);
    attrs.frame_addr = paddr >> 12;
    SET_PF(&pt[pt_idx], attrs);

    _flush_tlb();
    return 0;
}```
#

then how is it initializing

#

👻

#

page table poltergeist

bold grove
#

Well

#

I’m going out rn

#

So I will message later @small hound

bold grove
#

@small hound

small hound
#

yes

#

it has been a while

bold grove
#

im walking page tables like you said

#

let me show

small hound
#

👍

bold grove
#

what do you think

#

theres only 2 levels of paging

#

in 32 bit

#

mind ou

#

you

small hound
#

separate those out into functions

bold grove
#

okay bet

bold grove
small hound
#

for freeing

#

looks ok

#

👍

bold grove
#

@small hound i modularized this nicer

#
int map_page(uintptr_t vaddr, uintptr_t paddr, PageAttributes attrs) {
    vaddr = _align_down(vaddr);
    paddr = _align_down(paddr);
    if (vaddr == 0) return -1;

    uint32_t indices[2] = {_pd_index(vaddr), _pt_index(vaddr)};
    PTE* tables[2] = {RECURSIVE_PD, NULL};
    for (int i = 0; i < 2; i++) { // walk page table levels
        switch (i) {
            case 0:
                _pd_lvl(tables, indices, attrs);
            case 1:
                _pt_lvl(tables, indices, paddr, attrs);
        }
    }

    asm volatile ("invlpg (%0)" : : "a" (vaddr));
    return 0;
}
small hound
bold grove
#

sorry

#

bro

#

because you said

#

there should be a loop

bold grove
small hound
#

I forgot you're on 32 bit 💀

bold grove
#

so ive been losing my shit

small hound
#

it's usually a bit better on 64 bit

#

I mean if it works it works

bold grove
#

well

#

ive spent hours

#

so

#

idc

small hound
#

✅ great job

bold grove
#

heres how im doing ranges

#
int unmap_range(uintptr_t vaddr, size_t size) {
    size_t pages = _align_up(size) / PAGE_SIZE;

    vm_region_t *cur = vm_regions, *prev = NULL;

    while (cur) {
        // check if the range is within the current region
        if (cur->start == vaddr && cur->end == vaddr + size) {

            if (prev) {
                prev->next = cur->next;

            } else {
                vm_regions = cur->next;
            }
            break;
        }

        // move to next region
        prev = cur;
        cur = cur->next;
    }

    for (size_t i = 0; i < pages; i++) {
        uintptr_t va = vaddr + i * PAGE_SIZE;
        if (unmap_page(va) < 0) {
            // todo: handle error
            return -1;
        }
    }
    return 0;
}

int map_range(uintptr_t vaddr, uintptr_t paddr, size_t size, PageAttributes attrs) {
    size_t pages = _align_up(size) / PAGE_SIZE;

    vm_region_t* r = _make_region(vaddr, vaddr + size, attrs);
    if (!r) {
        return -1;
    }
    r->next = vm_regions;
    vm_regions = r;

    for (size_t i = 0; i < pages; i++) {
        uintptr_t va = vaddr + i * PAGE_SIZE;
        uintptr_t pa = paddr + i * PAGE_SIZE;
        if (map_page(va, pa, attrs) < 0) {
            // rollback if failed 
            // prevent mem leaks
            // todo: actual error handling
            for (size_t j = 0; j < i; j++) {
                uintptr_t undo_va = vaddr + j * PAGE_SIZE;
                unmap_page(undo_va);
            }
            vm_regions = r->next;
            return -1;
        }
    }
    return 0;
}
small hound
#

sounds great 😃

bold grove
#

heres the whole new thing

small hound
#

by the way mister flopmaster, my TLB eviction is kind of not the best way to do it. it's not exactly wrong but it also isnt the most efficient way

#

the better way would be to have a list (global or per core) of pages to invalidate, so that when you map/unmap a region, rather than waiting after each page, you add all pages to the list and send the IPI to start doing TLB shootdowns

bold grove
#

Okay bet

#

Should u just create like a queue of pages to be invalidated?

small hound
#

depends on how you wanna do it

#

or if you have it, you can use a deferred procedure call

#

and somehow get the other cores to run that DPC

bold grove
bold grove
small hound
#

you should make a unit and integration test suite to test these and such

bold grove
#

Yeah right now my resting functions are just little static functions in my source files

#

So in vmm.c I have some static functions

bold grove
#

Testing

#

My bad

small hound
#

oh ok

bold grove
#

Like

#

Vmm_range_map_test

#

That bullshit

#

But yes I do plan to make a testing suite at some point

#

I haven’t pushed this yet only bc git is stupid on my laptop

bold grove
#

It just gets stuck on “pushing”

small hound
#

internet issue?

bold grove
#

Nah

small hound
#

did you sign in with gh-cli

bold grove
#

Idk

#

Yes

#

I’m gonna figure this out rn

small hound
#

reboot it

#

everything fixes itself after reboot, trust

bold grove
#

So I can actually update this god forsaken thing

#

Said every windows user ever

small hound
#

for linux, be root, for windows, reboot

bold grove
small hound
#

still no flopperating system 😢

high hawk
#

i.e. operating systems for floppy disks!

bold grove
#

I wish i could run mine on one

#

Unfortunately multiboot is more bloated than a polish drunk dad

#

Why the fck do I need an EGA video mode

bold grove
#

Flopperating System

#

Changed thread name 🗣️🗣️

#

Just saying

#

I fucking love the buddy system

#

Doesn’t get easier than this

#

void* pmm_alloc_pages(uint32_t order, uint32_t count) {
    if (order > MAX_ORDER || count == 0) return NULL;

    void* first_page = NULL;
    void* current_page = NULL;

    for (uint32_t i = 0; i < count; i++) {
        void* page = NULL;
        for (uint32_t j = order; j <= MAX_ORDER; j++) {
            if (pmm_buddy.free_list[j]) {
                struct Page* p = pmm_buddy.free_list[j];
                pmm_buddy.free_list[j] = p->next;
                p->is_free = 0;
                p->order = order;

                while (j > order) {
                    j--;
                    buddy_split(p->address, j);
                }

                page = (void*)p->address;
                break;
            }
        }

        if (!page) {
            if (first_page) {
                pmm_free_pages(first_page, order, i);
            }
            log("pmm: Out of memory!\n", RED);
            return NULL;
        }

        if (!first_page) {
            first_page = page;
        }
    }

    return first_page;
}
bold grove
small hound
#

you can make a NUMA aware bitmap allocator

#

it will just be bad

bold grove
#

Doesn’t that have nothing to do with bitmap / buddy

#

I thought it was where memory access time depends on the memory location relative to the processor

#

It’s an smp thing iirc

small hound
#

non uniform memory access

bold grove
#

Yes

bold grove
bold grove
#

Bro my parents are crazy fucks

#

They destroyed my fucking laptop which I just fucking bought for uni

#

For no fucking reason too besides being fucking assholes

#

It’s because I didn’t give them 200$ out my paycheck

#

When I literally

#

Don’t even make 500 after taxes a now each week

#

So consider the project on hold

#

All of my source code is gone

#

Computer is fucked

small hound
#

🙏

bold grove
#

It’s just so fucking infuriating living with people like this

#

I got bread but god damn can I just have my laptop 😢

bold grove
#

Update: I bought a new computer

#

Setting it up later

#

Flopperating system may be back

#

Hopefully I can fucking push git commits too

bold grove
#

Okay we got it

bold grove
#

dude

#

it took me so damn long

#

to write this shit

#
PDE* vmm_clone_pd(PDE* _src_pd) {
    // allocate new page directory (zero ts)
    PDE* new_pd = (PDE*)pmm_alloc_page();
    if (!new_pd) return NULL;

    for (uint32_t i = 0; i < PAGE_DIRECTORY_SIZE; i++) { 
        if ((i << 22) >= KERNEL_VADDR_BASE) {
            new_pd[i] = kernel_pd[i];
            continue;
        }
        if (!_src_pd[i].present)
            continue;
        
        // define the src pt and create a new one
        PTE* src_pt = (PTE*)(_src_pd[i].table_addr << 12);
        PTE* new_pt = (PTE*)pmm_alloc_page();
        if (!new_pt) return NULL;

        _zero_pt(new_pt);

        for (uint32_t j = 0; j < PAGE_TABLE_SIZE; j++) {
            if (!src_pt[j].present)
                continue;

            // allocate new frame 
            void* new_frame = pmm_alloc_page();
            if (!new_frame)
                return NULL;

            // memcpy old frame to new frame
            void* old_frame = (void*)(src_pt[j].frame_addr << 12);
            flop_memcpy(new_frame, old_frame, PAGE_SIZE);

            PageAttributes attrs = {
                .present = src_pt[j].present,
                .rw = src_pt[j].rw,
                .user = src_pt[j].user,
                .frame_addr = ((uintptr_t)new_frame) >> 12
            };

            SET_PF(&new_pt[j], attrs);
        }


        PageAttributes pt_attrs = {
            .present = 1,
            .rw = 1,
            .user = 1,
            .frame_addr = ((uintptr_t)new_pt) >> 12
        };

        SET_PF((PTE*)&new_pd[i], pt_attrs);
    }

    return new_pd;
}
small hound
#

lmao what's with the constant code dumps

#

we have version management at home

bold grove
#

Just too lazy

#

Plus I’d rather show it here if it’s just short blocks

blissful grove
#

its alright, these threads are for sharing your progress

#

I reckon 50% of that is either kernel panics or code snippets lol

#

or screenshots of code 😠

noble remnant
#

beautiful rename

#

how fast does it flopperate though

bold grove
bold grove
bold grove
#

GitHub repo update: 3-5 days from now

#

Maybe tonight if I don’t drink a fuck ton but

bold grove
#

I just fucking suck at version control

#

And it kinda makes me mad

pliant oriole
blissful grove
#

Lol take your time, no one here is rushing you

bold grove
#

kinda cooking atm so

#

we wil see

small hound
#

oh yea a thing I found helpful was to encapsulate all globals that are used in a bunch of places in one struct like this

struct charmos_globals {
    char *root_partition;
    struct vfs_mount *mount_list_head;
    struct vfs_node *root_node;
    volatile enum bootstage current_bootstage;
    uint64_t core_count;
    struct scheduler **schedulers;
    struct core **cores;

    volatile bool panic_in_progress;
};

extern struct charmos_globals global;```
#

idk u might find it helpful

#

I found it very neat and useful

bold grove
#

Oh shit yeah

#

That’s not a bad idea

#

I should probably reserve a bit of memory in the kernel to hold a struct like that at boot

small hound
#

just

struct charmos_globals global = {0};
bold grove
#

anyways

#

i should invlpg only after unmapping, right?

small hound
bold grove
#

no

small hound
#

are you sure

bold grove
#

wait

small hound
#

if 0xfff000 is mapped to 0x4000, and you go and map 0xfff000 to 0x1000, you are indeed changing page table entries

bold grove
#

yeah so for mapping and unmapping

#

i need to invlpg

#

got it

small hound
#

bold grove
#

anywhere i change pt

#

i need to invlpg

#

well great, no paging fault now

#

okay yessir

#

got it

#

lemme show

#

how im mapping my memory

pliant oriole
bold grove
#
int map_kernel_space(uintptr_t vaddr, size_t size) {
    if (vaddr < KERNEL_VADDR_BASE) {
        return -1;
    }

    page_attrs_t attrs = {
        .present = 1,
        .rw = 1,
        .user = 0
    };

    uintptr_t phys_addr = (uintptr_t)pmm_alloc_pages(0, size / PAGE_SIZE);
    return map_range(vaddr, phys_addr, size, attrs);
}

int map_user_space(uintptr_t vaddr, size_t size) {
    if (vaddr >= USER_SPACE_END) {
        return -1;
    }

    page_attrs_t attrs = {
        .present = 1,
        .rw = 1,
        .user = 1
    };

    uintptr_t phys_addr = (uintptr_t)pmm_alloc_pages(0, size / PAGE_SIZE);
    return map_range(vaddr, phys_addr, size, attrs);
}
#

and the user gets 3gb