SnowOS

[hushed spire]

don't actually know how to go about this

[devout yarrow]

I think you are creating "threads" right now instead of "processes"

might be a bit of a misnomer to call them processes

just me tho

[hushed spire]

ah, mb

[hushed spire]

Eyyy!

now there's no context switching yet

so threads aren't actually running

But still, very cool

[devout yarrow]

huh

[hushed spire]

The scheduler is just printing some stuff out for debug purposes

[devout yarrow]

what does the printed stuff mean

[hushed spire]

just tell me which thread has been scheduled

[chilly birch]

these sound complicated but theyre super easy to implement iirc

[devout yarrow]

most trees aren't that bad

[chilly birch]

do you even know the formal definition of dimension

(thing i learned 1 month ago)

[quick hemlock]

https://cdn.discordapp.com/attachments/829403502349647952/1292284234265919488/attachment-36.gif

dont use an array for something like that

[hushed spire]

woooot is that

array for name makes sense but everything else is just... no

But yeah, arrays don't seem like they'd work very good for a thread/process list

[hushed spire]

Alright, let's see if I can figure out how to actually context switch now

Also now that I think about it, calling this function 'yield' might make more sense

// Decide which process to run next, and perform a context switch
extern "C" void schedule() {
    if (queueHead == nullptr || currentThread == nullptr) {
        kprintf(SCHEDULER, "Nothing to do.\n");
    } else if (currentThread->nextThread == nullptr) {
        currentThread = queueHead;
    } else {
        currentThread = currentThread->nextThread;
    }
    kprintf(SCHEDULER, "Running Thread: %lu\n", currentThread->tid);
    apicWrite(0xb0, 0);
}

[hushed spire]

Okay... that doesn't seem horribly complicated Eflags, the general registers and any data segment registers should also be pushed on the old stack and popped off the new stack. If the paging structures need to be changed, CR3 will also need to be reloaded.

I'm a bit confused about what I do when switching to a new task for the first time, the registers wouldn't be on the stack in that case no?

ah

well, I suppose the ISR pushes them onto the stack anyhow, doesn't it?

[devout yarrow]

you would create a dummy stack that has the registers on it, yes

the first switch would be a "trampoline"

treated differently from all other switches

(if you choose to go this route)

[hushed spire]

ahh, okay, that makes sense

[devout yarrow]

you really don't need to do the apicWrite in the schedule routine

this is more of the ISR handler's problem

ISR can call schedule and LAPIC EOI after schedule returns or whatever

[hushed spire]

fair point

[devout yarrow]

you can avoid needing a trampoline context switch in a bit of an interesting way (if you are smart about it)

I dont need to setup any fake stacks, because my scheduler starts off with the sched->current as NULL, and if the schedule() routine detects that sched->current is NULL, it doesn't save the current register context anywhere

so when my main calls scheduler_yield(), I don't have any running sched->current, and the scheduler effectively "forgets" about the register context it receives from main and just loads the next context and no trampoline is needed

ofc this is a flaw in the scheduler design because it currently saves RIP, CS, SS, etc. since it is only ever called from the interrupt handler. I'll need to adjust it to only save callee-saved registers upon manual yields, but this works for rn

[hushed spire]

huh, that's interesting

[devout yarrow]

so if you want your manual yield to not be called via the interrupt, you can have it only save callee-saved registers, but this means you need a "fake stack" and you'll need a trampoline so the first thread will have its "return address" (the entry point of the thread) on its stack

fun stuff 👍

[hushed spire]

Okie, lemme have a sec to wrap my head around that XD

[hushed spire]

I'm debating if I wanna do a kernel rewrite rn lol

I feel like a could do a few things a bit better than how I did them before

I'll at least get basic scheduling setup first tho

[devout yarrow]

you can make a unit test suite and rewrite individual parts and test them to make sure they behave as expected 😉

A tip: Make a custom linker section for tests. Then have symbols to define the start and end of the test section. you can have a struct test (align it to 64 bytes - i had weird issues when I didn't), and then place them in that linker section

then the test runner just gets the start and end of the linker section and runs through all tests in it

unit tests have let me do so so many rewrites with confidence that everything will continue to work

[hushed spire]

okie, thx for the advice!

[hushed spire]

New todo list! (Cuz it helps me lol)

[ ] Add Unit Tests for maintainability
[ ] Move SMP Initialization to start of kernel start-up (And debug issues that pop up)
[ ] Stop allocating virtual addresses so jankily (implement a basic tree for it)
[ ] Make a way of knowing if a threads being ran for the first time, and setup a trampoline stack
[ ] Actually figure out how to context switch
[ ] Figure out how you're planning on grouping processes and threads
[ ] Make threads & processes slowly get bumped down priority levels over time

[narrow lake]

What do you need a trampoline stack for? Or rather, how do you create your threads?

[hushed spire]

I need some way to be able to context switch into that thread no?

And if the stack is just... empty, kind of hard to do that

[quick hemlock]

you can put stuff onto the stack from your current thread. You really should only be switching between kernel threads (so kernel stacks), which are usually always mapped anyway

really you can setup the new thread's stack to look like it just called yield() from the start of the thread's entry function

so it'll restore the caller saved regs, then 'return' to the entry function and carry on execution as if it was always doing that

[open bluff]

do you mean saving/restoring the threads registers and such on interrupt? or on manual yield or ?

[hushed spire]

Okay, that's kind of what I was thinking I was supposed to do

[devout yarrow]

oh yeah a funny thing I do to get unit test test data is to do things like #embed a file with the bee movie script into a large string and use that as testing data

just a silly thing

and remember that if your stress tests don't break your code they are not stressful enough

[hushed spire]

Alright, lemme try to figure out how to add unit tests

[devout yarrow]

https://github.com/BlueGummi/charmos/blob/main/include/tests.h u can look at this, i hope it is understandable

[hushed spire]

ah, thx!

[devout yarrow]

and then just do stuff like this

REGISTER_TEST(ext2_stat_test, SHOULD_NOT_FAIL, IS_UNIT_TEST) {
    EXT2_INIT;

    FAIL_IF_FATAL(root->ops->create(root, "ext2_stat_test", VFS_MODE_FILE));

    struct vfs_node *node;
    struct vfs_dirent out;

    FAIL_IF_FATAL(root->ops->finddir(root, "ext2_stat_test", &out));

    node = out.node;
    TEST_ASSERT(node != NULL);

    struct vfs_stat empty_stat = {0};
    struct vfs_stat stat_out = {0};
    node->ops->stat(node, &stat_out);

    /* this should return something */
    TEST_ASSERT(memcmp(&stat_out, &empty_stat, sizeof(struct vfs_stat)) != 0);

    flush();
    SET_SUCCESS;
}```

shrimple

[hushed spire]

got it!

[hushed spire]

I feel like I need to be less incessant on doing things super correctly first go XD

It does me more harm than good and I end up getting nothing done

[devout yarrow]

ya that's a good idea, for example with your scheduler you can start with a UP RR to get the hang of things and then move to SMP RR and SMP MLFQ

that's what I did

and then remember to add synchronization primitives like semaphores, mutexes, etc

since you'll need those a lot later on

[hushed spire]

I also need to figure out how I wanna do syscalls (that can't be directly handled by the kernel)

split between having a separate server for handling that, or having all syscalls be the same (i.e. syscall occurs, kernel goes "oh, this is an open syscall, I can't directly handle this" ask the corresponding server to do it, and then hand stuff back to the process)

I would imagine the second method is slower

[devout yarrow]

I would guess that the first method is a bigger hassle tho since userspace ideally doesn't need to worry about such things

I would probably go with method 2 and make it transparent to userspace

[hushed spire]

true I suppose

[hushed spire]

Okie, we gonna get context switching working today!

(at least sorta working lol)

[devout yarrow]

how do you half-get context switching working ⁉️

i mean it's as simple as

; void switch_context(struct context *old, struct context *new)
global switch_context
switch_context:
    ; Save callee-saved regs into *old
    mov [rdi + 0x00], rbx
    mov [rdi + 0x08], rbp
    mov [rdi + 0x10], r12
    mov [rdi + 0x18], r13
    mov [rdi + 0x20], r14
    mov [rdi + 0x28], r15
    mov [rdi + 0x30], rsp
    lea rax, [rel .Lreturn_point]
    mov [rdi + 0x38], rax

    ; Load callee-saved regs from *new
    mov rbx, [rsi + 0x00]
    mov rbp, [rsi + 0x08]
    mov r12, [rsi + 0x10]
    mov r13, [rsi + 0x18]
    mov r14, [rsi + 0x20]
    mov r15, [rsi + 0x28]
    mov rsp, [rsi + 0x30]
    jmp [rsi + 0x38]     ; jump to saved RIP


.Lreturn_point:
    ret```

[hushed spire]

I don't trust myself mainly

[devout yarrow]

stress test

[hushed spire]

but yeah, should be simple enough

oh, huh, wait am I supposed to jump to the RIP instead of ret'ing to it (or does either work?)

I'll just do that for now since it seems easier lol

[devout yarrow]

just depends on what you wanna do

[hushed spire]

ah okay

[hushed spire]

gasp!

Now... threads other than thread D should be running, but still, awesome!

[devout yarrow]

this is a uniprocessor round robin right?

great start

[hushed spire]

ye

...hey wait a minute

; Save Registers
    mov [rdi + 16], rax
    mov [rdi + 24], rbx
    mov [rdi + 32], rcx
    mov [rdi + 40], rdx
    mov [rdi + 48], r12
    mov [rdi + 56], r13
    mov [rdi + 64], r14
    mov [rdi + 72], r15
    lea rax, [rel .returnPoint]
    mov [rdi + 88], rax

    ; Load new registers
    mov rax, [rdi + 16]
    mov rbx, [rdi + 24]
    mov rcx, [rdi + 32]
    mov rdx, [rdi + 40]
    mov r12, [rdi + 48]
    mov r13, [rdi + 56]
    mov r14, [rdi + 64]
    mov r15, [rdi + 72]

[devout yarrow]

bro copied me

[devout yarrow]

does this only take one argument

i dont think this is actually saving registers and loading registers

you just move things to and from the rdi pointer

void switch_context(struct context *old, struct context *new);```should look more like this

just double check that

[hushed spire]

it doesn't, it takes two, that's the issue lol

[devout yarrow]

uhhhh why does save/load only use RDI

load should use RSI here if your function signature matches mine

hmmmm

sus

[hushed spire]

idk why I did that XD

[devout yarrow]

there's your bug ig

s/rdi/rsi for the lower half

[hushed spire]

ah, I have realized another mistake lol

I switch contexts before doing apicWrite(0xb0, 0); so the timer interrupt never fires again

Ayyy, we getting somewhere

AYYY!!!

it did not like the while loop I had in the thread function lol

Now it would be good to figure out how to actually destroy the thread after it returns

seems like loops cause the scheduler to never get called again, that's certainly not supposed to happen lol

(until that thread returns)

[devout yarrow]

remember to not tie your scheduler to the interrupts

LAPIC EOI should be handled by the ISR

void isr_timer_routine(void *ctx, uint8_t vector, void *rsp) {
    (void) ctx, (void) vector, (void) rsp;
    LAPIC_SEND(LAPIC_REG(LAPIC_REG_EOI), 0);
    schedule();
}``` like this

[hushed spire]

ah, okie

[devout yarrow]

schedule() never does anything related to the timer isr stuff

[hushed spire]

also mb, seems like threads just get ran to completion for some reason

[devout yarrow]

do you have a reaper thread yet? my thread creation pushes the address of thread_exit onto the stack so when the thread returns it automatically is cleaned up

static void thread_exit() {
    disable_interrupts();
    struct thread *self = scheduler_get_curr_thread();

    self->state = ZOMBIE;
    reaper_enqueue(self);
    enable_interrupts();

    scheduler_yield();
}``` like so

uint64_t *sp = (uint64_t *) stack_top;
*--sp = (uint64_t) thread_exit;```

[hushed spire]

ah, okay, no I don't, that would probably help lol

[devout yarrow]

just have a simple reaper thread that you can wake to clean the threads marked to be reaped

simple stuff

and you can run it once a second just to be safe

void reaper_thread_main() {
    while (1) {
        bool i = spin_lock(&reaper.lock);

        struct thread *t = reaper.queue.head;
        reaper.queue.head = reaper.queue.tail = NULL;

        spin_unlock(&reaper.lock, i);

        while (t) {

            /* bye bye */
            t->state = TERMINATED;

            struct thread *next = t->next;
            thread_free(t);
            t = next;
            reaper.reaped_threads++;
        }

        thread_sleep_for_ms(1000);
    }
}``` super simple

[hushed spire]

Okie! Imma take a break and afterwards work one that

[devout yarrow]

oh yeah if things have bugs, don't get discouraged because the amount of fun things you get to do once you get a scheduler up and running is incredible, for example you can have worker thread event pools, and deferred procedure calls, and you can easily spawn a thread and just magically run a function with a function like thread_spawn(entry)

genuinely lots of fun (at least for me)

I would recommend you get the following facilities up and running after the scheduler

• reaper thread (super ez)
• blocking mutexes, semaphores, other locks you don't have now
• timer-triggered deferred function calls (so use the HPET or something to call a function in X milliseconds)
• condvars (conditional variables)
• event pools that can wake worker threads to execute functions in the event pool
• async IO interfaces for your block devices to use (stuff like generic request systems)

and anything else you find necessary

[hushed spire]

thx for the list! 👍

[hushed spire]

Okay ik I got context switching working, but I kind of want to try to get it to work how I was trying to do it

also I'm not swapping RSP, which would be helpful 🤣

seems rax, rcx, and rdx don't need to be saved

[hushed spire]

Well, that's certainly no good lol

[devout yarrow]

do you plan to do stack traces or nah

it helps a lot in debugging if you do live panic stack traces from the kernel

maybe also print what exception happened

probably page fault tho

[hushed spire]

ye I keep meaning to do this lol

seemed to stop faulting tho

now it just hangs

which is arguably worse

👀

well that's certainly not correct

rsp is also 0

okie fixed it, rsp and ss are what they're supposed to be now, still hanging tho

time to break out gdb

ah

seem to be stuck in a spinlock

nope nvm

[hushed spire]

WOOOT THE HECK HAPPENED TO THE PAGE TABLES?!?!

How does performing a context switch go this badly XD

I'm not even touching CR3

Well, at least that explains something

[devout yarrow]

it's not a race condition, the threads are playing 🎵 musical chairs 🎵

[hushed spire]

I feel like I needa do some more reading before I continue on with threads

in the meantime, I'm going to (try to) refactor some of my code

[hushed spire]

Ah great, SnowOS isn't starting on actual hardware, what'd I do?

[hushed spire]

OH FOR THE LOVE OF-

Why was I halting when Serial couldn't be enabled 😭

I swore I fixed that

well works now

except not really cuz I don't do anything when CPUID 0x15 can be used

oops, deadlocking lol

[devout yarrow]

deadlocking on real hardware?

but not in VM?

interesting

[hushed spire]

not really lol, the if statement for when I can use CPUID to get the frequency locked after the clause, but the function didn't unlock at the end (but it is unlocked in the clause when CPUID can't be used)

void setupInterval() {
    TicketSpinlock::lock();
    uint64_t frequency = 0;
    if (getCpuid(0x15).ecx != 0) {
        TicketSpinlock::unlock();
        kprintf(YUKI, "Getting Crystal Clock frequency...\n");
        TicketSpinlock::lock();
    }

    else {
        disableLvtTimer();
        apicWrite(LAPIC_DCR, 0b1011); // Div by 1

        apicWrite(LAPIC_LVT_ICR, 0xFFFFFFFF);

        uint32_t startCount = apicRead(LAPIC_LVT_CCR);
        hpetSleep(50);
        uint32_t endCount = apicRead(LAPIC_LVT_CCR);

        apicWrite(LAPIC_LVT_ICR, 0);
        uint64_t ticks = startCount - endCount;
        ticksPerNs = ticks / (50 * 1'000'000);

        TicketSpinlock::unlock();

        kprintf(YUKI, "The LAPIC ticked %d times in 50 ms\n", ticks);
        kprintf(YUKI, "LAPIC Ticks per NS %lu\n", ticksPerNs);
    }
}

never unlock unless the else clause is hit

[devout yarrow]

indentation is generally a dumb thing to complain about but here it would help a lot

do you indent your if-elses like this a lot

[hushed spire]

yeah?

do I do it weirdly?

(also arguably maybe I should remove the spinlock in kprintf)

[devout yarrow]

yeah it makes the else look like a separate piece of code rather than

if (thing) {
   a();
} else {
   b();
}```

does that cause readability issues for you tho

your style

if it doesnt then who cares

[hushed spire]

oh, looks pretty similar except the else comes after the if's closing bracket

[devout yarrow]

where do you find code that indents else clauses like yours btw

this is the first time I've seen this written this way

[hushed spire]

Uh, idk where I saw it. Usually I do it the way you're saying, i dunno why I did it differently there

[devout yarrow]

was that why u didnt catch the bug

[hushed spire]

the style is kind of all over the place sometimes lol, I need to fix it

[devout yarrow]

clang-format is a huge help

find . -name '*.c' -o -name '*.h' | xargs clang-format -i

works wonders

[hushed spire]

🤷‍♂️ that or I just wasn't thinking, probably the latter XD

[devout yarrow]

or '*.cxx' in your case or '*.cc' or '*.cpp' or any of the other millions of things C++ allows

[hushed spire]

lol, good to know

[hushed spire]

okie so, pretty sure I shouldn't be spinlocking in half my functions

and just lock before calling said function

anyhow

time to write an actual virtual address allocator

[hushed spire]

I presume the start of the tree would be the start of the higher half, since that makes finding addresses easy

If I need to allocate something in kernel space, I go right, need something in user space, I go left

Mainly confused about how I would build the AVL tree

[devout yarrow]

use ranges of addresses, no?

[hushed spire]

well, yeah lol

[devout yarrow]

do you not know how to implement a binary tree 🤨

do it in userspace first

[hushed spire]

what I meant was idk how to figure out if a VA is in use or not in any way that isn't slow

[devout yarrow]

wdym

[hushed spire]

obviously I don't wanna allocate a VA for something that was in use by something else. I'm just trying to figure out how to go about that, I dunno if it just isn't put in the tree or if I do put it in the tree and just mark it as used or something. And how I'm supposed to know that "Hey, address 0xcafebabe is in use, don't put that on the tree/mark it as used" in any way that isn't like, manually checking each address lol

[devout yarrow]

wait what are we doing here tho

is this for processes in userspace

i thought you made a kernel slab allocator

[hushed spire]

I need to be able to allocate VAs

[devout yarrow]

so like map addresses for a process?

[hushed spire]

e.g. something goes "Hey I want you to map this physical address into memory" the VMM needs to know what VA it can map that to

[devout yarrow]

ohhh you can literally use a bump allocator to do that

void *vmm_map_phys(uint64_t addr, uint64_t len, uint64_t flags) {

    uintptr_t phys_start = PAGE_ALIGN_DOWN(addr);
    uintptr_t offset = addr - phys_start;

    uint64_t total_len = len + offset;
    uint64_t total_pages = (total_len + PAGE_SIZE - 1) / PAGE_SIZE;

    if (vmm_map_top + total_pages * PAGE_SIZE > VMM_MAP_LIMIT) {
        return NULL;
    }

    uintptr_t virt_start = vmm_map_top;
    vmm_map_top += total_pages * PAGE_SIZE;

    for (uint64_t i = 0; i < total_pages; i++) {
        vmm_map_page(virt_start + i * PAGE_SIZE, phys_start + i * PAGE_SIZE,
                     PAGING_PRESENT | PAGING_WRITE | flags);
    }

    return (void *) (virt_start + offset);
}

[hushed spire]

that feels kind of scuffed tho 🥲

[devout yarrow]

static uintptr_t vmm_map_top = VMM_MAP_BASE;

nah this works fine

you get terabytes of memory

hundreds of terabytes

no need to overcomplicate it

[hushed spire]

and I was told by r4 to do it properly XD (or maybe that wasn't what he meant idk)

[devout yarrow]

#define VMM_MAP_BASE 0xFFFFA00000200000
#define VMM_MAP_LIMIT 0xFFFFA00010000000``` you probably won't be mapping physical -> virtual like this often, usually just for drivers

like how base address register give you a physical address

[devout yarrow]

i dont see how this would cause a problem though

[hushed spire]

🤷‍♂️ I just kind of want to do it properly lol

[devout yarrow]

I mean this is not problematic and thus can be considered "properly"

what scenarios will you encounter where you want to map physical to virtual?

it's mostly for device driver MMIO

which you typically do once at boot and never again

[hushed spire]

I mean sure

I can just do it later

Unless I run into issues

[devout yarrow]

didnt u say u had a problem with wanting to do things totally correct the first time and that slowed u down

this could avoid that

like it's not "Totally Correct™ " but it's "Functional and Works"

[hushed spire]

okie, I'll just write my AVL tree later

I should probably focus on threads atm anyhow

[hushed spire]

ESPECIALLY BECAUSE OF THIS NONSENSE!!!

I still dunno what's going on there

[devout yarrow]

does it consistently look like that or does it change aroudn

might be race condition

[hushed spire]

idk, I'd have to check

[devout yarrow]

anyways if u somehow do have some free time you can implement an AVL tree since it's nice to know how this stuff works and have working impls for it. you'll probably need it later on in development anyways for other parts of ur kernel

i think visualizing these is fun

now that you bring up binary trees i wonder how a binary tree would wear pants

would it be like one big pant over all the nodes or little pairs of pants for each node and its children.... hmmm

top 10 questions science can't answer

[hushed spire]

Okay yeah I think it's a race condition, lemme try actually giving each thread it's own pagemap like I'm supposed to be doing

[devout yarrow]

wait give each thread its own pagemap? wdym, like their own PML4?

[hushed spire]

sorry I meant process XD

[devout yarrow]

ohh ok

[hushed spire]

each thread doesn't have it's own pagemap lord

[devout yarrow]

per thread address spaces would go crazy

[hushed spire]

nope, still ain't working 😭

[hushed spire]

Alright, I can't with this, I need a break

[hushed spire]

okie

time to figure out why my context switch code is making my Page Tables have an existential fit lol

or I could just nuke my scheduling code and start over

which probably isn't half bad an idea

I'm partially suspecting my setupContext function

setupContext:
    mov rdx, rsp
    mov rsp, [rdi + 64]
    mov rax, [rdi + 72]
    push 0x10
    push rsp
    pushfq
    push 0x08
    push rax
    
    push 0x0
    push 0x0
    push 0x0
    push 0x0
    push 0x0
    push 0x0

    mov [rdi + 64], rsp

    mov rsp, rdx
    ret

Since the page tables only go all wack after iretq is called in contextSwitch

ignore the copy-pasted push 0x0 6x over lol

[hushed spire]

okie y'know what, Imma just nuke my scheduling code XD

I mean I doubt it'll fix it, but hey, who knows

wanted to refactor some things anyhow

and I have literally 0 clue of what the issue could be

[hushed spire]

Okay I'm putting my foot down

I refuse to continue until I get my interrupt handler telling me what exception occured

I have gone waaay to long without it XD

[hushed spire]

I couldn't think of a more creative nickname 😭

so I'm just gonna shill my own OS, very humble, I know

[hushed spire]

anyhow, kind of got this working now

[hushed spire]

WHAT?!

okay I can't with this

I'm gonna mess around with the initrd

and come back to this later

[devout yarrow]

the gdb in question:

you can remove ur panic handlers and just use gdb

gdb /home/gummi/charmos/build/kernel/kernel -ex "target remote localhost:1234"

this way you can do stuff like inspect threads, backtrace things, print variables, etc.

very fancy

[hushed spire]

'kay I'm just gonna quit for today, I'm too drained lol

I'll do some more tmrw

[quick hemlock]

You start the allocation with a region(s) of memory that you know are free. You set up the initial virtual memory space, so you know whats there and where.

If you're following what limine sets up, you can use the big gap above the hhdm up until the start of the kernel.

For userspace it's even easier: you can allow from 0x1000 up until the end of the lower half minus 1 page (see syscall issue on intel and I believe amd can try to speculatively execute non canonical addresses. That was something else, nvm)

No need to overcomplicate the allocator btw, a simple list of what's free/used will do.

[quick hemlock]

tbh thats not great, you do have loads of virtual address space... until you dont. Might be not a problem for most kernels here, but imo its a silly limitation, especially for such a fundamental resource.

like to bootstrap yourself sure its fine, but for a running system its pretty bad.

[hushed spire]

Ahh, okay that makes sense, thanks! 👍

Feeling much better and more motivated after last night (lets see how long that lasts lol)

[devout yarrow]

I'm guessing that this allocator is for mapping physical addresses to virtual ones tho, right? not a malloc implementation?

for my malloc-like thing (kmalloc) it's a proper slab allocator

[hushed spire]

not very, apparently 😭

maybe I should just take a break and go finish OSTEP

[hushed spire]

I may honestly just start over with my kernel lol

[devout yarrow]

how come

[hushed spire]

I dunno, kind of feel like the code is a mess 🤷‍♂️

[devout yarrow]

you don't even have that much code though

[hushed spire]

And I kind of speedran somethings lol

[devout yarrow]

you can pretty easily unmess it

[hushed spire]

true

[hushed spire]

felling a bit better now 👍

granted I said that... checks notes 6 hours ago so, maybe I should stop saying that aloud WAIT 6 HOUR-

[quick hemlock]

the conversation was about an allocator for virtual address space. My issue was with you suggesting to just leak resources because they are plentiful.

damn normally this channel scrolls down quite a bit when I reply, slow day for SnowOS

[quick hemlock]

lol we all have off days, nothing wrong with that.

[devout yarrow]

i mean, an allocator that maps physical addresses to virtual addresses is probably not used often. i chose my approach because that is terabytes of memory and there is not even 64 gigabytes of physical memory (on the machine i will run this on), plus that "allocator" would only be used during boot to map MMIO and things, but fair enough

I just didnt want mister BSD here to burn himself out from writing something unnecessarily complicated again

[quick hemlock]

maps physical addresses to virtual addresses
if you're just talking about a direct map, then yeah thats how you would do that.
For init-time stuff it also makes sense (I did say this earlier), but not for general purpose use.

[devout yarrow]

ah ok i think we got confused on what this was for then lol

I thought he already had a slab allocator for virtual memory

and this was just for direct maps

[hushed spire]

okie, time to get back in the groove

gonna move all the paging stuff to a seperate paging.cpp instead of being in vmm.cpp

[hushed spire]

I've been slacking recently, I need to stop lol. Anyone feel free to yell at me in here when I've gone to long without an update

Anyhow, made a silly mistake when moving paging stuff to a seperate source, so need to fix that

Since I have recently learned that putting definitions in a header is bad (I mean, I sorta knew that before), I might wanna change these to defines

constexpr uint64_t ptePresent = 0x1;
constexpr uint64_t pteWrite = 0x2;
constexpr uint64_t pteUser = 0x4;
constexpr uint64_t ptePwt = 0x8;
constexpr uint64_t ptePcd = 0x10;
constexpr uint64_t pteNx = 0x8000000000000000;
constexpr uint64_t pteAddress = 0x0000fffffffff000;

Eyy! All fixed!

[hushed spire]

Okie! Time to work on scheduling!

Let's hope this doesn't make me wanna jump off a cliff

[hushed spire]

Need to try and figure out how to make a circular doubly-linked list (since I think that's what I use for threads)

[devout yarrow]

it's just a doubly linked list but the tail->next = head and the head->prev = tail

there's no real reason to make it circular I just thought it would look better if my round robin was actually round

you can just enqueue at the tail and pop off the head to run a task

[chilly birch]

there is a real reason to make it circular

it makes enqueue and dequeue branchless

thats why circularly linked lists exist

[hushed spire]

gud to know!

[devout yarrow]

oh yeah

[wise ridge]

great opportunity to add a bunch of branches for safe unlinking and assertions :^)

[hushed spire]

Okay how the heck is this

setupContext:
    ; Save the old RSP and switch to the new one
    mov rdx, rsp
    mov rsp, [rdi + 8]
    hlt

causing all of this

can I not just mov something into rsp?

that seems like it'd cause a GPF or just not compile tho

Not this

well that sure as heck don't look right

okay what if I move the new stack into a different register first 🤔

well now I'm not triple-faulting, wut the heck is this tho

I'm gonna go on a whim and say it's not flanterm's fault lol

huh, okay moving anything into rax causes a panic

some I'm probably doing something else stupid lol

ah, clearing interrupts would probably help

how do I keep forgetting to do that

(I realize I probably have to do more than just cli to make sure I don't get interrupted but imma pretend that's good enough for now XD)

[hushed spire]

hrmmm

if (runningQueue == nullptr && readyQueue != nullptr) {
        kprintf(SCHEDULER, "No running threads, picking a new thread from ready queue OwO\n");

        currentThread = readyQueue->nextThread;
        readyQueue = currentThread->nextThread;

        runningQueue = currentThread;
        currentThread->nextThread = nullptr;
    }

I might use my first instance of a class in this kernel and chuck all the lists into a class

and then I can just call a function which will update the list instead of me doing it in schedule()

Might look a bit nicer

[devout yarrow]

why is there a queue of running threads (if that's what that means)

do you keep track of all threads currently running across cores in a queue

[hushed spire]

Don't you usually have a running queue?

[devout yarrow]

wdym

[hushed spire]

Like you have a Running Queue, a Ready Queue, a Blocked Queue, and then maybe a few others

[devout yarrow]

uhm you can do that if you want

but why do you need it

[hushed spire]

¯_ (ᵕ—ᴗ—)_/¯ i don't really know tbh

I'll worry about it later, rn I'm just trying to context switch without everything exploding lol

[devout yarrow]

also what's this logic for

if there's nothing to run u should just load the idle thread

idle thread can be as simple as a while(1)asm volatile("sti; hlt");

[hushed spire]

I did not know there was such thing as an idle thread lol, now I do (probably heard about it at some point)

good to know 👍

[devout yarrow]

you are adding the stack size to the pointer right

like if you say

stack = malloc(stack_size);
rsp = stack``` things wouldn't work because you'd need to say

```c
stack = malloc(stack_size);
rsp = stack + stack_size;``` or whatever

since it goes to lower addresses

[hushed spire]

yeah I'm doing that

I figured it out

[devout yarrow]

what's the current bug lol

[hushed spire]

Uhhhh, no obvious one's I can see, I mean context switching doesn't work properly, but I didn't really need it to, I was just trying to test something (also my logic for scheduling is sorta wonky lol)

Now time to go read a bunch so I actually know what I'm doing lol

We cook tmrw 🔥

[hushed spire]

(P.S. if anyone finds any, pls tell me so I can fix them)

[hushed spire]

Ahh, okie, that's interesting

Threads are picked from the current queue in
priority order until the current queue is empty. At this
time the next and current queues are switched. This
guarantees that each thread will be given use of its slice
once every two queue switches regardless of priority.

[devout yarrow]

"next and current queues are switched" is a bit of an odd way of putting it but yea ig that's correct

usually there's no need to "switch queues" you just have something like

struct sched {
    struct thread_queue queues[num_levels];
};```

and you just pick a different queue level to run

I suppose you could have some

struct thread_queue *current_queue;``` but that seems like extra work

[chilly birch]

no, you misunderstand it

[devout yarrow]

what does the book mean here

[chilly birch]

in an older version of ULE, each cpu had two sets of ready queues. newly enqueued threads would be placed on the "next" ready queues, and it would run threads from the "current" ready queues. eventually the "current" queues would empty out and then it would swap the pointers of the current and next queues

[devout yarrow]

huh that is very interesting

[chilly birch]

each set of ready queues would operate like the plain priority queues that youre referring to

yeah it was an anti-starvation mechanism

[devout yarrow]

ok very interesting to know

[chilly birch]

ULE no longer works this way

they got rid of this and replaced it with the timeshared vs interactive (real time) queues

[hushed spire]

Yeah instead of doing this

Thread *readyQueue = nullptr;
Thread *runningQueue = nullptr;
``` I think I'll just have one queue for now, since there's not really a point to having these two at the moment

[chilly birch]

there is a point

ULE didn't do that for no reason, it accomplishes something important

[hushed spire]

there is a point, just not at this moment lol, since I don't really handle the two queues differently, I am gonna do it that way, imma just use a queue for now, and when I get that working I'll do it with two

(or however many queues I should have)

[chilly birch]

you dont handle them differently

[chilly birch]

also this isnt what it is to begin with

its two completely separate SETS OF ready queues

so for example if you have 32 priority levels, youll have 32 queues, and you select threads from the highest queue that contains threads or whatever

in this scheme, you will have TWO SETS OF 32 ready queues

so 64 in total

when a thread is placed on the ready queues, it goes on the "next" set of ready queues

when you select a thread, you look at the "current" set

when the current set runs out, you swap it with the next set

and start consuming threads from that

this prevents indefinite starvation of low priority threads by high priority threads

because a high priority thread wont immediately get re-enqueued to a high priority queue and get immediately selected again

itll get put on the "next" set of queues and wont get run again until the "current" set has emptied

so every lower priority thread gets to run first

[hushed spire]

that's a lotta queues O_O

But yeah I am planning on doing it that way, I'm just trying to do a really simple RR to get started

[chilly birch]

this is basically as simple as round robin

its like 1 extra thing

on top

[hushed spire]

ah okie, well that doesn't sound too bad

[hushed spire]

ooooh, we getting somewhere

I think the whole "page tables getting obliterated" thing I was dealing with is due to the thread being preempted in the middle of something important, because if threadA just calls hcf(), we're fine (well, maybe not fine as you can see, but the page tables are fine), but if I try to print something in it, then everything goes outta wack

which y'know, when I don't save rdi and rsi...

ye that's probably gonna happen lol

now... I have to actually figure out how to go about saving rdi and rsi

since switchContexts also takes those as arguments

lesgo!

okay now I need to figure out how to actually save rdi and rsi

hmm, maybe not so "lesgo" lol

don't think that's what RSP should be =/

welp, imma take a lil break for now

[hushed spire]

I wonder if I should do variable time-slices (っº - ºς)

they're kind of just fixed to 50ms atm

[devout yarrow]

what would you use it for

[hushed spire]

seems to be used in priority

[devout yarrow]

how so

[hushed spire]

how big your time-slice is seems to depend on your priority (the higher your priority, the bigger the time-slice, lower priority, lower time-slice. Or wait, was it the other way around 🤔)

obviously not doing that atm was just wondering

[devout yarrow]

ok you could keep it planned for the future probably, do note that doing something like an mmio write to the LAPIC is a comically computationally costly operation, it takes about 3,000 clock cycles to literally write 4 bytes to the LAPIC timer register thing

at least for me

[devout yarrow]

you can also consider the idea that having more runnable threads at once would warrant a reduction in the timeslice (from 10ms maybe to 2ms or such), whereas less threads would have longer timeslices each

[hushed spire]

2ms still seems way too small to me lol

[chilly birch]

this probably has the opposite effect of what youre hoping for

[hushed spire]

ULE seems to have a minimum time-slice of 10ms

[chilly birch]

if you have that many runnable threads then the cpus are oversubscribed and youll only prolong that situation by spending so many extra cpu cycles switching tasks at a higher frequency instead of actually doing work and letting them finish

they are also compute-bound rather than interactive so responsiveness doesnt really matter much. so its better to let their priority decay (to timeshared in the case of ULE) and anything where responsiveness matters will be marked interactive and go on the real time queues and preempt them

[devout yarrow]

https://github.com/torvalds/linux/blob/909d2bb07dc0e08ea81841f7c901f0f16f965f0e/kernel/sched/fair.c#L77 linux seems to do 0.75ms

[hushed spire]

O_O

[chilly birch]

thats 750 microseconds

not milliseconds

youre off by a factor of 1000

[devout yarrow]

huh

isn't 0.75ms equal to 750 microseconds

idk

[chilly birch]

somehow i totally misread what you said

carry on

[devout yarrow]

https://elixir.bootlin.com/linux/v6.5/source/kernel/sched/fair.c#L741

seems like linux (back in CFS days) had a strategy that would set a time period in which all the tasks get to run at least once

static u64 __sched_period(unsigned long nr_running)
{
    if (unlikely(nr_running > sched_nr_latency))
        return nr_running * sysctl_sched_min_granularity;
    else
        return sysctl_sched_latency;
}
``` seems like it takes this to get the time period

and

```c
slice = __calc_delta(slice, se->load.weight, load);``` does this to find the optimal timeslice for each task

where `__calc_delta` does this

<https://elixir.bootlin.com/linux/v6.5/source/kernel/sched/fair.c#L323>

or in sane people math terms

delta_exec * weight / lw.weight``` or

(delta_exec * (weight * lw->inv_weight)) >> WMULT_SHIFT```

pretty neat

and __calc_delta uses the Fun and Whimsical strategy of scaling stuff to avoid floating point

I have no idea what EEVDF does and I have not looked at it yet

[hushed spire]

thx for the info! 👍

now imma go read some more lol

(and try to read for more than like, 10 minutes this time)

hopefully I can get context switching working today

[hushed spire]

Honestly I may just take a break from this project and come back when I feel more equipped to handle it

Cause I really seem to have 0 clue what I'm doing lol

If anyone knows any reasonably challenging C++ projects I should try, feel free to let me know

[hushed spire]

woooow, I actually got an issue on my repo lol

[marble cave]

lmao that was me

[hushed spire]

oh that's funny lol

@flint burrow if you have any other issues with building SnowOS feel free to tell me here, might be easier to help you this way

[flint burrow]

Ok!

[hushed spire]

Okie I tried drawing up a little diagram of how SnowOS would work for funsies, very subject to change of course but I hope it at least makes a little sense lol

[hushed spire]

I'm now realizing that literally everything above Yuki is userland technically, but I think we got what I mean 😅

[devout yarrow]

ring 1 and ring 2 💔

[hushed spire]

Well, wouldn't really count as a microkernel then, would it XD

[hushed spire]

I am now thinking about how I would go about this without making it suck lol

[devout yarrow]

you're gonna have a blast with IPC

Trust No One

[hushed spire]

That does seem like it'd be the case 🤣

One would figure that an NT-esque design would translate well to a microkernel

[hushed spire]

Okay, I think I'm pretty sure I can put most, if not all, handle management into a userspace server

Can't really why the kernel would need to get involved

[hushed spire]

I'd imagine it'd work something like this?

[hushed spire]

Okay genuinely, I may just switch to a monolithic kernel

Cause I am not confident in my abilities whatsoever to make things not incredibly slow lol

And I'm probably already giving myself enough work with doing a custom API and allat, I don't know if I need to deal with the difficulties of writing a microkernel on top of it

and I think a monolithic kernel makes more sense for this project anyhow

very sad for all the microkernel fans, I know 🥀

Also in case anyone's curious, I'll probably start development again in November, so we'll see how much reading I get done before then :3

[marble cave]

Stop with redesigning everything 💔

you can do it

[hushed spire]

Wot, I haven't redesigned everything XD

I mean, I am going to, but that's a separate matter

I was mainly winging it at the start, I have a slightly better idea of what I want to do now

[marble cave]

Monolithic unix like kernel with mlibc and limine

[hushed spire]

I feel like mlibc would get very upset at my OS lol

[blazing epoch]

NO

[mental gulch]

%0

[hushed spire]

how the heck does the repo have 18 stars lol

anyhow, haven't read any OS books in a hot minute

I'll read OSTEP some before I go to sleep

[hushed spire]

Might just go ahead and start working on this again once I get a decent ways through Windows Internals

I mean I haven't finished OSTEP

[blazing epoch]

reactos the second?

[hushed spire]

But I can probably wait to read the FS stuff for when I actual start doing FS stuff

[blazing epoch]

just wing it

[hushed spire]

oh god no

[blazing epoch]

basedn't

[hushed spire]

lol

no Windows compatibility unless I by some miracle end up porting wine

which is unlikely

[blazing epoch]

make kernel level wine

boom

reactos the second

[hushed spire]

[blazing epoch]

behold penultimate jank

drunkOS

because wine

😭

[hushed spire]

I think my OS will end up being jank enough as is XD

[blazing epoch]

lmao

[hushed spire]

Okay so, I'm extremely behind when it comes to school (loved my mom, but part of me kind of wishes she put me in public school), sooo, while I play catch-up, this won't be worked on as much as before, I'll still work on it (still planning on restarting the project in November, after I've gotten a decent ways through Windows Internals and the FreeBSD Book), just not quite as much since I'll be focusing a lot more on school work.

Hopefully we'll return to our regularly scheduled OS autism soon 🙃 hope y'all understand and have a nice day!!! :3

[blazing epoch]

👍

[hushed spire]

it's over halfway through October and I've still barely read any of Windows Internals 💀

reading is not my strong suit evidently

[hushed spire]

WE BACK!!!

Now setting up my env took too much outta me so I'll actually do stuff tmrw

[hushed spire]

...

:3

okie

time to add the most important feature: "GDT Init... OK"

also I wanna change the font lol

[visual walrus]

remaking your os?

[hushed spire]

Ye

[visual walrus]

i'm rewriting my allocator a bit, lol

[hushed spire]

coolio

[visual walrus]

bc it was a bit of a mess for usermode

[hushed spire]

wut

[hushed spire]

vro 🥀

does -fcoff only generate 32-bit code or what

[hushed spire]

passing -fwin64 instead gives me this

maybe I should just go back to CMake 😆

although this probably isn't Meson's fault lol

probably

like why the heck does lld-link think a label is a undefined symbol

ah hah!

think I fixed it

[hushed spire]

and we got interrupts back

[hushed spire]

I have awoken

Probably gonna work on the interrupt handler today

ah yes I also need to implement printf

[void elm]

Is idt really a part of hal

[hushed spire]

I mean, it's an Arch specific thing no?

wouldn't all the CPU init stuff go into Hal?

[halcyon star]

Idt init... Oopsies

[hushed spire]

aight now I can work on the interrupt handler.

[hushed spire]

Okie I am stoopid, how the heck do I pass the interrupt frame to my handler routine

oh, uh, that's kind of right, but it's in reverse

what if I just order the struct in reverse to account for that?

ayyy!

[hushed spire]

:3

[hushed spire]

Alrighty, going to work on a timer interrupt today

Now I wonder how the heck I'm supposed to calibrate the timer without vmmMapPhys() 😅

[hushed spire]

Also moved all the Limine stuff into Hal since I plan to make this bootloader agnostic later (and by that I mean it can boot using Limine or SnowBoot and nothing else )

[hushed spire]

had to go finish my samurai costume lol

anyhow, uh, right, timer interrupt

I think Imma have to mull about it cause I have 0 clue how I'm going to go about this lmao

[hushed spire]

Uhm, a quick cursory look at React's source would lead me to believe, no, interrupt stuff is supposed to go into Ke 😅

[vivid helm]

In NT (and thus in ReactOS) interrupts belong to Ke, KINTERRUPT, but imho, it is a matter of personal views whether to delegate it to Ke or Hal. One might argue, that it's definitely a subject to "hardware abstraction".

[visual walrus]

depends on which timer

[blazing epoch]

2/2=4

[hushed spire]

haven't been on discord much as of late lol

[hushed spire]

I should uhhhh, probably do something

Imma do something very obviously productive and change the font

(and also add serial logging)

I should be using C++23, so I'm pretty sure I can just use #embed for the font

[hushed spire]

ah wait, it may actually be in C++26 🤣

[hushed spire]

okay so, scratch that, I'll do this some other time because I was cooking, and it took a lot outta me lmao

[blazing epoch]

fire

[blazing epoch]

C--

[hushed spire]

mmm, yes, remember when I did OSDev

I, uhhhh, need to work on this more

still haven't read Windows Internals 🤣

It would probably be a good idea to finish that before coming back to this, buuuuut, eh, idk

I feel like I'm more motivated to actually read OS Books if I'm actively doing OSDev

I'll probably do some warming up before coming back to this though

like I fogor half the things I read in OSTEP lmao

[blazing epoch]

reading is lame dont do that

[hushed spire]

So I got the second to last tooth on my bottom jaw on the right side removed a bit back because cavities (sucks that I got better about brushing my teeth after it was already a bit past saving), which sucks cause the one on the opposite side also fell out a while ago, so all in all, life been kind of sucking lately. So I'm gonna cope like how all normal people do and get back to OSDev >:3

I might just work one things in the normal order instead of the one I heard will talk about because it's easier for my head to handle 🙃

[hushed spire]

Also might move the project over to codeberg

[hushed spire]

SnowOS has a website now!!! >:D

there's literally nothing on it tho lol

I just wanted to go ahead and get it setup

probably will post devlogs and such to it in the future

snowos.org if anyone's curious

[hushed spire]

Alright, gonna go ahead and make a simple freelist allocator real quick

I'm just gonna use the limine memmap structure for now, and make a general one later

[hushed spire]

Yikes I'm rusty

Got it done though

I should take a looksie at how React does it

but anyhow, will probably work on the PFNDB tomorrow

[hushed spire]

I feel so overwhelmed like there's a hundred, 800 page CS books out there I gotta read, and I have to learn this, and that, and it feels like it's gonna take 10 years for me to write an even slightly competently written program

anyhow, rant over lol

no PFNDB today, don't really feel up for it at the moment

[hushed spire]

Honestly I might just go on hiatus again until I feel more comfortable with this kind of stuff

And actually read my OS books, for like, realsies this time

take it one step at a time

Year of the SnowOS desktop coming... in 2035

I'm sorry, I know I keep delaying this project lol (even though I doubt anyone is super invested in it)

I might try making a small UNIX-like OS in the meantime

just to be implementing stuff while I read OSTEP

And I've been told to do that before jumping headfirst into making a non-UNIX OS

[hushed spire]

small update, I've been working on a revised project doc, very much a W.I.P but I figured I'd post it here anyhow https://docs.google.com/document/d/1MfCyIjULQh93YQgmkt-WZilyvDlnxrmsqsHGQgcsJfM/edit?usp=sharing

[hushed spire]

Since snowboot doesn't have it's own thread, imma yap about it here :3

I should probably figure out what I want the config file's syntax to be

I also need to figure out the boot protocol

maybe something like this?

[hushed spire]

#1471581492797771817 message

So yeah I think imma go back to working on SnowOS

I think I need to work on the PFNdb rn

[hushed spire]

oh

actually, I should probably work on a serial driver first

[hushed spire]

I was wondering why I was triple-faulting when an exception happened

I accidentally got rid of the line that inits the CPU lol

[hushed spire]

also slightly improved exception handling

[hushed spire]

Oh boy

I did not consider the debugging implications that came with using PE as my executable format lol

[hushed spire]

was a bit busy

anyhow

lemme at least try real quick to figure out how I can debug this 😭

[hushed spire]

I guess I just... won't debug for now?

seems like a slight issue lol

oh well

that's what I get for not using ELF

on the other hand

if I needed anymore convincing to write a small debug shell for this, I think I just got it lmao

It's weird cuz like

LLDB can read the symbols just fine

and it will set a breakpoint

but when I continue

nothing happens =/

and not like nothing happens because I set a breakpoint

my kernel just keeps going along as if nothing is wrong in the world 🤣

oh well

guess it's a problem for future me

anyhow

Serial driver time!!! >:3

[hushed spire]

Serial works!

[hushed spire]

I'll get started on the PFNdb later

after that I'll probably work on the paging stuff and then after that I'll try to work on SMP

[hushed spire]

aight, PFNdb time

shouldn't be too hard

from my understanding it's pretty much just an array

First lemme study React's source some

what is this elusive <ntoskrnl.h

huh?

there's three of them? 😭

I'm guessing I want ntoskrnl/include/ntoskrnl.h

uhmmm, if I can figure out where React creates the PFNdb

[hushed spire]

I'm refactoring my code to use NT-style type names

I'm sure everyone will love that

[hushed spire]

okie, finished with that :3

honestly I think I'll do the PFNdb tmrw

[hushed spire]

so much stuff to do lol

and this is just like, broad strokes

I still need to figure out what else I need to do

I'd say it's a pretty decent to-do list so far though, so I might just figure it out along the way lol

[hushed spire]

hmmm, I wanna add stack unwinding (that's what's called right? ) to my panic function

anyhow, Imma go do some math, read, and then we work on SnowOS

[rare plume]

is it time for the third part yet?

[hushed spire]

yes :3

gonna work on initing the PFNdb

Just gonna take a chunk outta the memory map

also think I might consolidate all the MM initialization into on Mm::Initialize() or something. Instead of having seperate ones for the pmm, paging, vmm, etc.

[rare plume]

a whole bunch of terms i don't (but probably should) understand :3

[hushed spire]

lol

oh

before that I was gonna change up the logging a bit

[hushed spire]

Wrote a function to log to serial ^w^

dunno if I wanna keep the whole [??] part

looks a bit odd with the file right after it

ye I think I'll just remove it

okie there we go

time to for PFNdb!

probably gonna suck

but oh well

Pretty sure all I have to do is get the amount of usable pages I have, multiply that by the size of an entry, and that's how many bytes I need to store the array

minoca also uses a union?

iirc Linux also uses a union in it's struct page

[void elm]

a union

it has a billion unions there

[hushed spire]

oh

[void elm]

https://elixir.bootlin.com/linux/v6.19.8/source/include/linux/mm_types.h#L79

[hushed spire]

what's with all the unions?

[void elm]

a page can have many different states and subsystems currently owning it

[hushed spire]

I'll guess I'll just put a free field in the entry struct for now, and add more as I need them?

[void elm]

yea

[hushed spire]

I may have procrasinated a wee bit

anyhow

[rare plume]

relatable

[hushed spire]

I need to finish the init code, and preferably, make it not atrocious

[rare plume]

mine is like less than 40 bytes, though there's probably a lot to add. anyways, off i go to my os thread

[hushed spire]

This is currently for-loop galore rn

which uh

I get the feeling is bad

I can probably do the PFNdb init and the freelist init at the same time

oh yeah, I also wanted to make the HAL a DLL

like the cool kids do

also think I need to work on IRQs

[hushed spire]

Actually it might be useful to keep track of whether or not a page is paged or nonpaged pool

gosh there's so much I don't know 😭

wait

hol' up

I need to init the freelist after the PFNdb don't I?

cuz otherwise I'm gonna end up adding the PFNdb's pages to the allocator

I think that looks a bit better?

boy things do go a lot faster if I don't doom about how I'm probably doing X wrong

Okie, lemme put the freelist init after the PFNdb init now

Since the PFNdb is already created, it should be pretty trivial for the freelist to get itself setup after that

Actually, I can just embed the freelist into the PFNdb can't I?

is there a reason I wouldn't wanna do that?

hmmmm, I'm thinking

Okie so it seems like I need an early allocator and all that jazz if I wanna setup my PFNdb properly

we work on that tmrw

hopefully won't be too hard

wait, array is a freestanding header?

I just happened to be looking through freestnd_cxx_hdrs and saw it

Not really sure if I'd wanna use it, but it's neat that it's there :3

also somewhat inspired by ilobilix and wondering if I wanna try using modules or not

[hushed spire]

Guess I should probably work on this now huh?

I need to figure out what I'm doing for the bootstrap allocator

what if I just grab the largest chunk of the memory map and bump allocate pages

I put the trollface there but honestly this seems fine

don't think the bootstrap allocator needs to be too fancy

[hushed spire]

okie that was fairly simple :3 https://github.com/UtsumiFuyuki/SnowOS/blob/main/yuki/source/mm/early_alloc.cpp

okie so I have a bootstrap allocator

now I need to an early way of mapping physical memory

what if I just do another bump but instead for virtual addresses instead of physical ones

[hushed spire]

I got side tracked :3

anywho

The current conundrum I'm facing is I need to map my PFNdb (duh) which obviously means I need paging stuff, but I dunno how to tell the pfndb what pages have already been allocated (for the page tables)

[hushed spire]

another day, more of me attempting to write my OS (and, swiftly failing, probably )

[hushed spire]

oh yeah, need to write a separate function for panic so I can call it outside of the exception handler lol

[hushed spire]

okie, I think that looks fine

ah, actually setting KernelPml4 would uh, probably help

uhhh, hol' up, that ain't right

oh

I forgot to ensure that PhysicalAddress and VirtualAddress were page aligned lmao

well, the flags get set properly now anyways

[hushed spire]

oh wait, I think I see the issue here

yup! fixed it!

[hushed spire]

I'm finally drawing my mascot :3

[hushed spire]

finished!

now I should probably uh, actually work on my OS

[blazing epoch]

this guy better fix your VMM

or unguy

https://cdn.discordapp.com/attachments/1423054245430694053/1423055104952504450/cat_bomb.gif

is there even a vmm here im having a stroke

[hushed spire]

not yet no

I am working on paging rn tho

[hushed spire]

I should set a goal for the end of this week, I dunno what a reasonable goal would be though

Maybe get to scheduling

Not necessarily have the Scheduler written

Just get to a point where I can write the Scheduler

[hushed spire]

Maybe I should move the bootstrap stuff into a separate KeInit or something? And then have that call KeMain

[hushed spire]

wait I said I was gonna play smash with my brother when I got back from the park

okie after that we cook

I would really like to get the PFNdb setup today

[timber canopy]

cute!

[hushed spire]

aight, it's time >:3

[hushed spire]

okay, I should have everything I need to actually make the PFNdb now, now I just have to figure out how to implement it

time to drag out the old paper and pen

[hushed spire]

hmmm, okay I think I got an idea...

seems uber slow though

oh

well I thought of an idea to make it faster

okay well basically

I'll allocate a page for the PFNdb initially, then I'll go through the memmap and see how many pages need to be allocated/mapped for the current entry, then I'll just memset that area of the PFNdb to an initial state for an entry, and then just continue until i've entered all usabe pages into the PFNdb

seems simple enough

ofc maybe my idea is some grave OSdev sin idk lol

[blazing epoch]

i feel like having a "database" in a kernel in general is an osdev sin 😭

[hushed spire]

Windows has one, pretty sure Linux does too (struct page, at least I presume functions similar) so I'd think not 😅

[blazing epoch]

i guess calling it a database is the problem

feels deranged

[hushed spire]

just a tad messy

oh well, I'll refactor later

haven't even enacted the memset part of the plan and it's already slow 😭

maybe lemme try without logging

oh

yes much faster lol

[hushed spire]

then I'll just memset that area of the PFNdb to an initial state for an entry actually, can I do this with memset, I don't think I can

hmm

I think I'll continue this tmrw

I'll try to find someone else's impl of this because I feel like mine is very... bad XD

After this I'll implement spinlocks so I can get started on SMP

I also might think about implementing qspinlocks

[hushed spire]

ehh, okay yeah this doesn't seem that slow, I think it was just the 50 million logs it was printing to serial

[narrow storm]

I calculate a region of memory I need to store the info of all other regions. I then store that in the first region (that’s large enough)
And then all regions have the PFNdb pages for them carved out of themselves. Then to do a lookup I search for the region containing the proper range

This only works with an HHDM and isn’t as fast as other methods

But it dosnt need anything else to boot strap itself

I TECHNICSLY even have it up before the PMM

This seems somewhat similar to what you’re trying so?

[hushed spire]

ah

doesn't the PFNdb need to be sparsely mapped though? How does that work out with your method?

[narrow storm]

I just rely on the HHDM

And then pages part of the PFNdb are not tracked by the PFNdb

[hushed spire]

ahhh, I just chose some random VA-range above to kernel to start mapping at

[narrow storm]

I also only use regions larger than a certain amount

And later il gather them up into a separate sub stricture for very fragmented memory to use in last resorts

Eg memory for OOM killer to execute?

I call them parent/child/sister tables

[narrow storm]

No valloc

https://cdn.discordapp.com/attachments/1434603991957897258/1483237320978202925/image.gif

Though for this a fixed VA is fine if your doing virtually sparse PFNdb

I didn’t do that so I can always have a PFNdb to track all allocations

As I do that to ensure I don’t do double allocations or double frees as I track that state

[hushed spire]

okie dokie, good to know! :3

[narrow storm]

The only downside is that it’s O(n) for finding the child table, but in most cases in reality how fragmented is the memory map to make it slow down that bad

[hushed spire]

alright, enough procrastinating, time to learn about radix trees 😔

[hushed spire]

oh ye, lemme clean up the paging code

[hushed spire]

okie finished that up

[hushed spire]

okie, lemme see if I can implement a simple radix tree in C++, seems simple enough

though I will admit I am somewhat confused as to how you would represent a PFNdb as a radix tree

[narrow storm]

You could index the radix tree similar to how virtual memory does it

Input a physical address

And it ends up going to a table containing the data

[hushed spire]

[narrow storm]

You know how you take a virtual address and split that up into many chunks?

To index into the tables

[hushed spire]

Ye

[narrow storm]

You compute the size of that you need for the physical address space

And then set it up into table structure

And then fill them in

Potentially lazy too

And I presume that works because you don’t need to track about pages you don’t free and only use for that

Though I very well could be wrong™

It’s not a horrid way from what it seems

And very well a page could describe its self

[hushed spire]

ahh, okay

[hushed spire]

took a quick cursory glance at the Page Frame Database section in Inside Windows NT, and I can confirm that it seems like the freelist is embedded in the pfndb

anyhow I should uh

probably go to sleep

it's 12 AM for me

I'm also been further confused because it seems like Windows does just use an array for the pfndb

which is what I was initially trying to do

so like, which is it? 😭

[narrow storm]

There are many diffrent ways you can go about this

That’s the fun part

Though having the PMM embedded into questionable

My freelist is embedded into the free pages themselves and I lazily allocate the nodes and keep track of where I am

[hushed spire]

i mean idk maybe it isn't and I'm misintepereting it

There's at the very least a free list in the pfndb, idk if it has anything to do with the allocator tho

[narrow storm]

I mean NT does use a freelist allocator for the PMM so

[quick hemlock]

nah its a perfectly fine use. If you dont have a pfndb your only option is to use the freespace (this is also fine).

[narrow storm]

I’m just not sure how to execute somthing like that

Oh no yeah it makes sense

[quick hemlock]

pages and pfndb entries are a one to one mapping, you put the list linkage in the db entry instead of the free space of the page 🙂

[narrow storm]

Depends on how you set up the thing right

Before or after the PMM etc

[quick hemlock]

also remember that a pfndb entry has an ownership of some kind, the layout of the entry depends on who is using the page. If the page is free, the db entry has a certain layout, if its in-use, it has another layout. I also have a layout for when the page is used by page tables to store the count of in-use PTEs for a table.

[narrow storm]

Using a union id presume?

[quick hemlock]

abusing unions*

but yes 🙂

https://elixir.bootlin.com/linux/v6.19.9/source/include/linux/mm_types.h#L79

[hushed spire]

okie, I think Imma just go the route of having the pfndb just be an array. Maybe I'll change it later if I decide I think a radix tree would work better (I'll still mess around with radixs though cuz the seem fun :3)

[narrow storm]

I do wonder if I setup the PFNdb after the VMM to do a virtually sparse setup how one should handle the pages you had to allocate to set that up?
Woild hand waiving it away be fine considering you won’t free those pages? (Part of the root page tables and part of the PFNdb it’s self)

[quick hemlock]

for bonus points consider making it virtually sparse, only back the virtual ranges you need.

[hushed spire]

yeah this was my plan ^w^

[narrow storm]

This would also permit lazy allocation of PFNdb too iirc?

[quick hemlock]

I consider those pages part of the system init, they're just magically gone.

same with idle stacks (for me), per cpu storage and a few other things.

[narrow storm]

Yup makes sense since you never really need to free them anyways

[hushed spire]

ooh, good point lol

[hushed spire]

Okay the plan for today is pfndb and hopefully a little work on SMP

[hushed spire]

hmmm, how should I set the initial entry state for all the entries?

iterating through the pfndb seems like it'd take a while

[hushed spire]

okay pretty sure everything gets allocated and mapped correctly

now I just need to figure out how to actually set the default state for all the entries

[hushed spire]

so this is faster than I thought it would be :3

[hushed spire]

okey before I got to sleep Imma try to work on the freelist allocator some :3

pretty sure I can just have a "next" member in the PFN_ENTRY struct, and then build the freelist from that

[hushed spire]

also my PFNdb init does, in fact, not work correctly

[hushed spire]

okay I think it works now? 😭

[hushed spire]

Well actually freelist allocation isn't finished, I got busy trying to fix the pfndb lol

[hushed spire]

okie Imma work on my math and then try to get the freelist allocator into a usable state

then I'll work on percpu stuff and smp

[hushed spire]

Okay I think allocating pages works properly now

imma write a spinlock impl real quick! :3

if uh, I can figure out how to use std::atomic_compare_exchange

[void elm]

make a ticket lock

it doesnt require cmpxchg

well only for try_lock maybe

[hushed spire]

ah okie :3

yikes uh

I don't think that's right

is it cuz I'm using clang?

[hushed spire]

ate mah food

back to implementing spinlocks

[hushed spire]

not really sure what the percpu struct should have

[lunar vortex]

everything

[hushed spire]

and what precisely would "everything" entail?

I guess I wanna store the LAPIC id maybe?

You also generally have percpu thread queues iirc?

[lunar vortex]

as much as you want

some scalable designs rely a lot on replicating state per-CPU

[hushed spire]

ahh

[hushed spire]

I also need to work on getting keyboard input so I can work on the debug shell

[blazing epoch]

are you doing USB

[hushed spire]

uh, eventually yes :p

[blazing epoch]

yeah fair enough

people (one person mainly) had problems with usb on real hardware

idk

ok ps/2 tho?

[hushed spire]

from my understanding some hardware (laptops mainly iirc) internally use ps/2 for the keyboard. So yes, I need to write a PS/2 driver 😔

can't be too hard

[blazing epoch]

shouldnt be

i wrote a ps/2 mouse driver

somewhat unrelated but its not that difficult

[hushed spire]

coolio

[blazing epoch]

for PS/2 you really just need an idt and your pic set up

idk anything about the apics im sure you can do it there if you really want to

you also need a table with scancodes ig

[hushed spire]

speaking of which, I have page frame allocation, and cursed virtual address allocation, pretty sure I can use the early table setup stuff from uACPI now

[blazing epoch]

big scary

peak os: starts then shuts down computer

[narrow storm]

@hushed spire did you get that spinlock workin?

[narrow storm]

███████ os: starts and █████ your ████████ to do a ██ ████ █████

[hushed spire]

Yup! ^w^

Needs a bit of tweaking but seems to work just fine

[narrow storm]

amazing :3

is it the code i gave you?

make sure to licence it

/hj

[hushed spire]

It's pretty much just a slightly refactored version of my old impl actually

[narrow storm]

lolll

may i see?

i need to make sure its micro optimized

[hushed spire]

VOID Ke::SpinlockAcquire(PSPINLOCK Lock)
{
    __asm__ volatile ("cli");

    while (true)
    {
        if (!__atomic_exchange_n(&Lock->Flag, 1, __ATOMIC_ACQUIRE))
        {
            return;
        }

        while (__atomic_load_n(&Lock->Flag, __ATOMIC_RELAXED))
        {
            __asm__ volatile ("pause" ::: "memory");
        }
    }

    __asm__ volatile("sti");
}
``` Like I said still needs so tweaking, mainly not blindly disabling and re-enabling ints, but seems to work

[narrow storm]

this isnt gonna work lol

🥀

return means it never enables IRQs again

and you need to save and restore in most cases

and not have a sti at the end anyways

[narrow storm]

for critical sections*

[hushed spire]

oh yeah I noticed that don't worry

[narrow storm]

https://git.evalyngoemer.com/evalynOS/evalynOS/src/commit/e1c29eb7c8979af94a1b3e97429d4a790b026389/src/kernel/utils/spinlock.h

just steal this lol

it works XD

[hushed spire]

nooooo, I wanna make it myseeeelf

[narrow storm]

you still are

you have to rewrite for C++

[hushed spire]

[narrow storm]

this time you just have to slap a comment

also if you have bugs

you can yell at me now

[narrow storm]

I just feel bad looking at this lock impl

🥀

[hushed spire]

owch

[narrow storm]

Sorry

[narrow storm]

Also make them static inline

And just inside a header file

And maybe a .c for unlock if you need to call from asm

Because these are very tiny code wise and it’s to inline them for performance since you may use them a lot

and it means one less TU for the linker etc

[hushed spire]

tbf it looks like my impl is 90% of the way there I just need a way to check if interrupts are enabled or not

[hushed spire]

ah okie 👍

[narrow storm]

global interrupts_enabled
interrupts_enabled:
    pushfq
    pop     rax
    shr     rax, 9
    and     rax, 1
    ret

This is how I do it

I think I should use EAX on these though (besides the pop iirc)

Ah no eax should work

pushfq is 4b

Nvm I miss read

[hushed spire]

seems simple enough

[narrow storm]

[narrow storm]

It basically says. Put the flags register on the stack. Shift the IF bit and mask it

And return

As rax is the sysv return register

[hushed spire]

yaaay I got it right!

[hushed spire]

ah, I need to check what this would be for me since I don't think I'm following ths SysV convention

[narrow storm]

you should follow sysv

its way nicer then windows