SnowOS

[thorny plover]

Well, accept for the MMIO that I need... Limine maps this as reserved for some reason

[hushed spire]

Basically what I meant was, I was wondering if UEFI changes the allocated region's memory type to EfiReserved, which based on the memory map I'm getting I would presume that, no, it doesn't (which I suppose makes sense, don't really see why it would change the memory type)

So I'm wondering if my function is screwing with the page tables in some way

Cuz again, my page tables seem to get all messed up when I try to call initPmm()

Yeah I think that's the issue

Because if I just ignore 0xB0000000-0xBFFFF000 then it works (well okay I'm still page faulting but I think it's just a math error lol)

[hushed spire]

Well, I got it to work (or at least not page fault) veeeeeeerry hackily

But I need to think of an actually good way to make it work lol

[quick hemlock]

this is one of those 'you'll find it in the manual' moments

afaik you're only supposed to use the memory map from just before the last call to exit boot services (since that might fail and/or allocate lol)

[hushed spire]

Time to rethink how I'm gonna initialize the PMM lol

Big Problem 1 is that I need to figure out how to tell it to not touch the areas where the page table/kernel/memory map are located

I could just memcmp the entire page to see if it's empty

That may work

Although I don't think the entirety of a free page is actually 0

[narrow lake]

keep in mind that memory is not zeroed at boot

[hushed spire]

ye I realized that lol

[quick hemlock]

Bruh

Use a memory map

[patent cloud]

"hmm how will I make it know where memory, such as the memory map, is?" 😭

[hushed spire]

UEFI doesn't seem to mark allocated pages as not free in the memory map (ofc I could be wrong)

Which is obviously a problem cuz I don't wanna be touching allocated pages, but again, I would be all to happy to be wrong lol

[quick hemlock]

You are wrong, that would be pretty weird if it didnt

Yeah, AllocatePages() asks you what you want to mark the memory as after it's been allocated

When it finds a physically contiguous block of pages that is large enough and also satisfies the allocation requirements of Type, it changes the memory map to indicate that the pages are now of type MemoryType.

MemoryType being an argument to AllocatePages

[hushed spire]

OHHHHHHHHH, okay yeah, I think I misunderstood what MemoryType was used for lol

[hushed spire]

Okay so I allocate important stuff [that kernel needs] as EfiReservedMemoryType instead of EfiLoaderData and it works now

Well it's doesn't PF at least, I need to check if the logic is actually correct

[hushed spire]

I apparently have a multiplication overflow somewhere (don't ask me what happened to the line number lol)

[thorny plover]

Holy shit lol

If your rolling your own printf() it's likely something your doing there...

[hushed spire]

Tried making some ASCII Art for my kernel panic (it's supposed to be a ghost lol)

[hushed spire]

It's fine XD

[hushed spire]

Anyhow, need to figure out where the heck I have a multiplication overflow (or rather why, I think I know where considering there's only 2 multiplication operations in that entire source file lol)

Alright fixed it (used an int instead of a uint64 lol)

[thorny plover]

That'll do it 😅

Thank God for ubsan lol

[hushed spire]

agreed lol

Okay... it's working... I think. Meh, ig I'll find out later if it's broken or not XD. Seems to be working correctly though.

[thorny plover]

Yeah it looks fine...

[hushed spire]

Allocating a page seems to work! 😄

[hushed spire]

I wonder how I syscalls work with a microkernel, obviously most of the stuff is in userspace so it wouldn't really be a "syscall" perse would it?

Anyhow, seems like a (simple) PMM basically boils down to "allocate a page and free a page"

so ig I could get started on the VMM now

unless there's something else I should do first?

[hushed spire]

Mmm, okay Imma stay up a bit late working on my VMM (I won't stay up too late don't worry lol)

So I need to figure out what exactly a VMM actually does

a VMM seems a lot more complicated than a PMM lol

I suppose I'll just start with implementing functions for mapping and unmapping pages (which I can probably just steal from my bootloader)

[thorny plover]

It's not too bad... the VMM handles a hell of a lot more than the PMM for sure though...

https://github.com/ProfessorLongBeard/HelixOS/blob/v0.0.1-dev/kernel/mm/vmm.c

This is mine, but be warned it's not great lol. It's for ARM64, but it's pretty similar for x86

[hushed spire]

Okay there must be a better way to use objects for things than what I'm doing XD (I just keep using extern)

[thorny plover]

You'd basically handle remapping the kernel, etc, and other stuff

[hushed spire]

oh, thanks!

[thorny plover]

Wait, in your PMM?

[hushed spire]

Uh, I believe the only thing the PMM source externs is the Terminal

But like, the Terminal is an object, the PMM is an object, and the VMM is an object

Which I'm pretty sure is fine but I just think the way I'm going about it sucks lol

[thorny plover]

Well, it's your design, so you do what you feel is good for you haha

[narrow lake]

huh, do you keep a single page table for everything combined?

[hushed spire]

Uh, think I need to have the bootloader pass the kernel's physical address to the kernel so it can remap itself

[hushed spire]

So I'm just gonna do things normally instead of using a bunch of objects everywhere

in a bit

my brain needs a break lol

[hushed spire]

Alright time to make everything a normal function now

[hushed spire]

Think I finished finally

[hushed spire]

Ah, now printing is broken again

I'm taking a break lol

[hushed spire]

alright, fine, I guess I should fix this now XD

Also rethinking if I actually wanna do a microkernel or if I'm just gonna do a monolithic kernel

I mostly chose it at the start just cuz it sounded cool lol

But I (or atleast I'd like to think) know a bit more about OSes now, so I'm reconsidering

Ehh, I think I'll stick with a microkernel

Just thought I should consider if I actually wanted to use one or not

I think I like microkernels better though

[thorny plover]

Microkernels can benifit from better performance, etc, but comes at the cost of moving most things to userspace. That can get a bit tricky to handle haha

[hushed spire]

Printing to the screen should not be giving me this much trouble -_-

Okay you know what, I'm keeping my Terminal object for now

I'll probably change it later but I don't feel like dealing with it rn

I suppose I could just make kprintf store a buffer and then use the terminal to print that buffer (so I don't have to keep using extern everywhere I wanna print stuff)

I also need to make it so my terminal can scroll at some point

[hushed spire]

Alrighty, tomorrow I'll work on the VMM (and possibly PMM if it breaks lol)

[narrow lake]

More like generally worse performance, but better robustness for failing parts

[hushed spire]

Hmm, silly me forgot that my PMM is supposed to return phsycial addresses lol

Yay! We're page faulting! But that's good this time! (I also really need to make my interrupt handler tell me what interrupt actually happened lol)

[hushed spire]

Okay I have to actually think about this a bit lol

I can't read the start of the head to get the next page because the page isn't mapped

And it's not like I can just apply the HHDM (since the VMM is gonna change the mappings, so that wouldn't work after that)

Maybe I'm making this more complicated than it needs to be

[narrow lake]

You‘d very likely want to map a HHDM too

[hushed spire]

Oh, okay, so I can just apply the HHDM to access the page?

[narrow lake]

Yeah

[hushed spire]

ah okay, that makes my life easier lol

[narrow lake]

remember that this is your kernel, you can do what you want

[hushed spire]

Seems to be working so far

Also debugfs is saying that the kernel is trying to use 17 blocks, so I should probably figure out how to load indirect blocks now (my bootloader is just loading the first twelve blocks of the kernel into memory at the moment)

[thorny plover]

Yeah, anything 48K+ will end up needing to have indirect blocks handlled...

[hushed spire]

So... I think an indirect block just points to a block with a bunch of block ids as it's contents

Yeah I think that's all it is

So it shouldn't be too hard to deal with

[hushed spire]

Think I got it working

[hushed spire]

Hmm, okay I'm triple faulting when attempting to reload the page tables in the kernel

Imma take a break for now though

I suppose this is bound to happen when this mess is my init function XD

void init_vmm(std::uint64_t hhdm, std::uint64_t kernel_paddr)
{
    kern_terminal.kinfo(VMM, "Initializing VMM...\n");

    hhdm_offs = hhdm;
    pagemap.levels = 4;
    pagemap.top_level = pmm_alloc();
    pagemap.top_level = reinterpret_cast<std::uint64_t *>((std::uint64_t)pagemap.top_level + hhdm_offs);

    map_pages(hhdm, 0x0, 0x3, 0x100000000);  // Make sure to keep the HHDM mapped
    kern_terminal.kinfo(VMM, "Finished HHDM map! Remapping kernel...\n");

    map_pages(kernel_virt, kernel_paddr, 0x3, 0x10000);
    kern_terminal.kinfo(VMM, "Finished remapping kernel! ");
    kern_terminal.term_print("Kernel physical start 0x%x mapped to virtual start 0x%x\n", virt_to_phys(kernel_virt), kernel_virt);

    kern_terminal.kinfo(VMM, "Identity Mapping Page Tables...\n");
    map_pages((uint64_t)pagemap.top_level - hhdm, (uint64_t)pagemap.top_level - hhdm, 0x3, 0x10000000);
    kern_terminal.kinfo(VMM, "Finished Mapping Page Tables! Located at ");
    kern_terminal.term_print("0x%x\n", virt_to_phys((uint64_t)pagemap.top_level - hhdm));

    kern_terminal.kinfo(VMM, "Reloading CR3...\n");

    pagemap.top_level = reinterpret_cast<uint64_t *>((uint64_t)pagemap.top_level - hhdm);

    loadPageTables((uint64_t)pagemap.top_level);

    kern_terminal.kinfo(VMM, "Finished Initialization!\n");
}

[thorny plover]

[hushed spire]

Hmm, perhaps I should just rethink these functions instead of copy&pasting them from my bootloader lol

[thorny plover]

I agree..... 😅

[hushed spire]

Well, flags aren't being set correctly for some reason (should be r/w not just r), but other than that seems to be working

For some reason it was trying to access 0x00000000bfe966c8 idk why but I just id mapped it so it would stop page faulting

[narrow lake]

you always have the RIP and addr2line, look it up

[hushed spire]

I'm going to see if I can figure out how to get my exception handler to dump the registers

I should also probably just do some code clean-up

[thorny plover]

You'd want an assembly stub that creates a stack frame (e.g. pushes the registers to the stack), and just pass that as the first argument to your handler

[hushed spire]

Okay, I assumed I had to do something like that

[thorny plover]

Yeah, it's really simple

[hushed spire]

Remember when I said this? ...Yeah, I might've lied XD. Maybe thinking of making a new name, ig I'll think about it while I go work on Shogi++ since I need more C++ practice lol

AquaOS just sounds kind of bland to me personally

[thorny plover]

Which ever name you choose I think will be a good name ^-^

[hushed spire]

thankie!

Huh, just learned about range based loops (thx CLion lol)

[hushed spire]

I may just write a buddy allocator

I was gonna use one anyways so I might as well do it now

[thorny plover]

No harm in doing so

[hushed spire]

I might rename this to SnowOS

cuz I like snow & winter

[hushed spire]

Now I'm going to go back and reformat my code, because I flip-flop between styles sometimes, and that's probably going to byte me in the butt at somepoint

[hushed spire]

I'm almost done lol

[hushed spire]

SnowOS

Renamed it again lol

last time I promise

Also reformatted the code and restructured the repo

So I should be using camelCase for variables and functions now

[thorny plover]

Whenever I read old code that someone wrote that has no spacing between things I end up with the worst OCD...

[hushed spire]

I don't actually know how quick a buddy allocator is

I might keep the free-list allocator if it's faster

(And doesn't make my life super hard)

Although I should rewrite my init function, cuz it's kind of scuffed

[patent cloud]

not such a bad thing

do whatever keeps you motivated to continue with the hard stuff I guess :)

[hushed spire]

Was messing around with the ISR and made INT 60 invoke a syscall, not that I need them right now or anything, just did it for fun.

[thorny plover]

Niiiiice

So.... I fixed those issues I was having... turns out it was a virtio driver bug...

Needed to make the status volatile, fix a check, and relax the CPU while waiting for a status change....

[hushed spire]

Yay! Good job! 🎉

[thorny plover]

Thank you ^-^

I knew the driver had bugs, but completely forgot about them XD

[hushed spire]

I think I can just do this instead of creating a pointer just to access the page

*reinterpret_cast<uint64_t *>(hhdm + next) = (next + 0x1000);

[thorny plover]

I'm not very familier with C++ 😅

[hushed spire]

Y'know what now that I think about it this isn't really an issue is it?

the VMM will have already been setup when the mappings are changed

so I could just let the PMM page-fault and have the VMM handle it

(and I can just map the HHDM)

[thorny plover]

From what I've seen that's kind of what other folks do. Just have a page fault handler that maps the new page as needed, then continue execution as needed

[hushed spire]

Actually I couldn't lol

Infinite loop

works for other things just not the PMM

[hushed spire]

Okay this doesn't look that different than what I had before, but I think it looks a bit better.
https://gist.github.com/BlueSillyDragon/85f5945ce28de7de81686a4cc9e00fa5

If anyone knows a better way to do it, then feel free to tell me

Oh it doesn't work... lol

I'll figure it out later, Imma take a break

[quick hemlock]

This looks overly complex for a free-list init, I'll have a proper squizz when I'm at a PC next.

There's a lot of casting around pointers happening there, looks like a lot of room for error.

Also I can't unsee snOWOs now lol

[hushed spire]

you're right XD

[hushed spire]

Hmm, okay I'll try to do this differently in a sec

[quick hemlock]

so yeah for the descriptors I'd suggest getting rid of the descriptor size idea: use an array of pointers or just tightly pack them. You can also invert the 'is free memory' check.
If you want to save some time during init you can store add a count for how many pages are in the freelist entry (instead of assuming just 1), then you can start with a single freelist entry per memory map region - and this'll break down into single pages overtime.
Not mapping above 4G is another unnecessary complication imo, not sure why you dont map all usable memory - that removes another edge case.

[hushed spire]

Good to know! yeah the whole "not mapping above 4GB" is something I did in my bootloader and never changed lol, so I'll probably go back to make it map all of usable memory

[hushed spire]

So my bootloader should map all of memory to the HHDM now

EFI_MEMORY_DESCRIPTOR *desc;
uint64_t nop = 0;

for(uint64_t i = 0; i < getEntryCount(); i++)
{
desc = (EFI_MEMORY_DESCRIPTOR *)((uint8_t *)memoryMap + (i * getDescSize()));
nop += desc->NumberOfPages;
}
mapPages(pagemap, hhdmOffset, 0x0, 0x3, (nop * 0x1000));

And.... that made my PMM work? For some reason? It doesn't GPF now at least

Although it doesn't seem to work on HW now (or with --enable-kvm)

so I gotta fix that

[hushed spire]

Okay I changed it to this instead

EFI_MEMORY_DESCRIPTOR *desc;
        uint64_t nop = 0;

        for(uint64_t i = 0; i < getEntryCount(); i++)
        {
            desc = (EFI_MEMORY_DESCRIPTOR *)((uint8_t *)memoryMap + (i * getDescSize()));
            if(desc->Type == EfiConventionalMemory || desc->Type == EfiLoaderData || desc->Type == EfiLoaderCode || desc->Type == EfiBootServicesData || desc->Type == EfiBootServicesCode)
            {
                mapPages(pagemap, hhdmOffset + desc->PhysicalStart, desc->PhysicalStart, 0x3, (desc->NumberOfPages * 0x1000));
                nop += desc->NumberOfPages;
            }
        }
``` Which... still doesn't seem to work on HW, but at least --enable-kvm works now lol

[hushed spire]

Alrighty, time to figure out why this ain't working

I know my bootloader isn't great, but I wish I knew why it wasn't great so I could fix it lol

also wow, using --enable-kvm -M q35 makes QEMU much faster lol

Huh, weird

Works fine on a Dell Optiplex 3040 I have but not on the HP Pavilion I was testing it on

[thorny plover]

Figures lol

[hushed spire]

Also works just fine on my Raider GE68 HX (Which is much newer than that Pavilion mind you)

Sooo, idk what's wrong

Obviously a me issue

But idk what that could be (isn't an issue with memory, seems to run fine with 8GB of RAM)

I mean I have confirmation it runs on real hardware, so I could just deal with it later

[hushed spire]

I'm going to make CMake generate an .img file I can just run under QEMU instead of me needing to do my whole USB thing lol

[hushed spire]

Finally done! I can generate a properly partitioned .img now (took my way too long to figure out that gdisk could be ran non-interactively with sgdisk)

[thorny plover]

Did you know sgdisk is basically able to do this too? It's gdisk, but scripted... Here's an example usage if how I partition my disk in a shell script:

sgdisk \
    --new 1:0:+512M \
    --typecode 1:"ef00" \
    --change-name 1:"EFI" \
    --new 2:0:+ \
    --typecode 2:"8300" \
    --change-name 2:"Rootfs" \
    ${DISK_IMG} >/dev/null 2>&1

It's super useful!

[hushed spire]

Yeah this is what i was talking about lol

took me way too long to learn that existed

[thorny plover]

Lol it's such a helpful tool!

[hushed spire]

My PMM could definitely use some work, but it (seems to) work, so I'm going to work on my VMM some, and then I'll come back to it

[hushed spire]

Hmm, okay ubsan's saying I have a misaligned pointer here

head = *reinterpret_cast<uint64_t *>(hhdm_offset + head);

Which I suppose shouldn't be that surprising

This does look kind of sus

[hushed spire]

Also made SnowBoot create a custom memory map and pass that to the kernel

[hushed spire]

Ooh, this is making my head hurt lol

Making a free-list allocator probably shouldn't be this hard

I don't really know what's wrong, seems like it should work =/

[thorny plover]

I feel the same about fixing my VFS code lol

[hushed spire]

Alright, y'know what, I'm figuring it out tomorrow

[thorny plover]

Sometimes it's always best to take a break, and come back to it...

I've had to do it to be honest...

[devout yarrow]

uhm

[thorny plover]

Lmfao

[hushed spire]

Yeah lol, I would fix this but idk why it does that

[devout yarrow]

scandalous bug

[hushed spire]

Time to try and figure out why my PMM doesn't work

Maybe it just doesn't like this?

uint64_t *test = reinterpret_cast<uint64_t *>(bootInfo->hhdm + pmmAlloc());

Seems to work otherwise =/

nah nvm, still doesn't work XD

[hushed spire]

Perhaps I should find someone else's implementation of a Free-list allocator to take a gander at

[narrow lake]

Well, how does it fault or how does it ‚not work‘

[hushed spire]

Basically I can do something like

uint64_t test = pmmAlloc();
``` but if I try to typecast that to a pointer and add the hhdm (to, y'know, actually access the page), then it GPFs (which is weird because it seems like allocation works fine, as well as setting the head to the next page

I mean idk if the GPF is caused by something weird I did when I setup ubsan

cuz it seems to happen whenever ubsan complains about something

I wish I knew why ubsan was complaining about a misaligned pointer. This doesn't looks misaligned to me

uint64_t *pHead = reinterpret_cast<uint64_t *>(hhdm_offset + head);
head = *pHead;

[hushed spire]

and nvm it's complaining when I try to do this now lol

Lemme remove ubsan rq to see if that's the issue

[narrow lake]

step there in GDB and check if the things you set are the things you expect

[narrow lake]

how did you setup ubsan? It should not really cause any fault

[hushed spire]

probably an issue with my printf implementation tbh

[narrow lake]

oh lmao, you just have undefined behaviour or something then

[hushed spire]

(if it is ubsan causing the fault)

[narrow lake]

i'd just plug in https://github.com/charlesnicholson/nanoprintf or some printf like that

[hushed spire]

Ah okay, yeah I'll just do that (I think I learnt the basics of how printf works now anyhow)

[hushed spire]

Uhh, wuh oh XD

[narrow lake]

you can disable Floatting point

I have it setup like this:

#define NANOPRINTF_IMPLEMENTATION
#define NANOPRINTF_USE_FIELD_WIDTH_FORMAT_SPECIFIERS 1
#define NANOPRINTF_USE_PRECISION_FORMAT_SPECIFIERS 1
#define NANOPRINTF_USE_FLOAT_FORMAT_SPECIFIERS 0
#define NANOPRINTF_USE_LARGE_FORMAT_SPECIFIERS 1
#define NANOPRINTF_USE_SMALL_FORMAT_SPECIFIERS 1
#define NANOPRINTF_USE_BINARY_FORMAT_SPECIFIERS 1
#define NANOPRINTF_USE_WRITEBACK_FORMAT_SPECIFIERS 0
#define NANOPRINTF_SNPRINTF_SAFE_EMPTY_STRING_ON_OVERFLOW 1
#include <nanoprintf.h>

[hushed spire]

okay, that solved the first two errors! 😄 (still complaining about an undefined reference to __ubsan_handle_function_type_mismatch for some reason tho)

Okay I just tricked it into compiling by copying __ubsan_handle_type_mismatch_v1 and changing the name lol

Should figure out how to actually fix it though

[hushed spire]

Aah, time to get back to the grind, I've somehow screwed up implementing nanoprintf so I need to fix that 😅

[hushed spire]

Okay I was drawing for a bit, now I'm actually gonna fix it

Seems to be an issue with how I'm wrapping npf_snprintf()

[hushed spire]

I literally see no issue with this lol

void kprintf(char *string, ...)
{
    char buf[256];
    std::va_list arg;
    va_start(arg, string);
    npf_snprintf(buf, sizeof(buf), string, arg);
    kernTerminal.termPrint(buf);
    va_end(arg);
}

[hushed spire]

I have a little desktop pet to make OSDev that little less bit painful! Unfortunately, it only works on Windows lol

[hushed spire]

Seems like passing a va_list to a function expecting ... makes weird things happen (according to a google search, anyways)

Soooo, how the heck am I supposed to pass variable parameters to npf_snprintf XD

[quick hemlock]

You use the version that accepts a va_list lol

[hushed spire]

Oh, okay, weird...

[quick hemlock]

It just mirrors the stdlib versions

Look at vsnprintf

[narrow lake]

do you have non-kernel programming experience with C/C++?

[hushed spire]

Not a ton but yeah, mainly GameDev

[hushed spire]

Been feeling kind of unmotivated lately

Mainly because anything I do seems to break immediately lol

[hushed spire]

Also I might just use Limine for now and come back to SnowBoot at a later point

Because my hacky bootloader probably isn't doing me any favors

[hushed spire]

Or maybe not depending on how much of a pain it's gonna be to get my setup to use limine lol

[patent cloud]

what's wrong with it?

[patent cloud]

oh sorry was looking at this

[patent cloud]

that is a good plan :)

[hushed spire]

Eyy! Booting with Limine!

GDT & IDT no longer works though, so gotta fix that lol

[hushed spire]

I'm getting a page fault at 0xffffffff80001860 with 3 as the error code, which means there's a problem with the P flag and W flag (I think)

Well, the Present and Write bit are certainly not set

idk why it's trying to write to there though

[hushed spire]

It might be this?

gdtr    dw 0
        dq 0

gdtr is at 0xffffffff80001860

[hushed spire]

Okay just doing this

__asm__ volatile ("lgdt %0" :: "m"(gdtr));
``` fixes it

problem is I clearly don't know how to do this lol

that's definitely not correct

size looks fine but that's not the base of the GDT

oh, adding __attribute__((packed)) fixed it

[hushed spire]

sigh nothing ever works 🤣

I'll deal with this in the morning

Serial logging soon 🙏

[hushed spire]

Alright, time to figure out why this ain't working

No clue why, looks perfectly fine

[devout yarrow]

one second

wait mb

static inline void outb(uint16_t port, uint8_t value) {
    asm volatile("outb %0, %1" : : "a"(value), "Nd"(port));
}

static inline void outw(uint16_t port, uint16_t value) {
    asm volatile("outw %1, %0" ::"dN"(port), "a"(value));
}

static inline void outl(uint16_t port, uint32_t value) {
    asm volatile("outl %0, %1" : : "a"(value), "Nd"(port));
}

static inline void outsw(uint16_t port, const void *addr, uint32_t count) {
    asm volatile("rep outsw" : "+S"(addr), "+c"(count) : "d"(port) : "memory");
}

static inline void outsb(uint16_t port, const void *addr, uint32_t count) {
    asm volatile("rep outsb" : "+S"(addr), "+c"(count) : "d"(port) : "memory");
}

static inline void outsl(uint16_t port, const void *addr, uint32_t count) {
    asm volatile("rep outsl" : "+S"(addr), "+c"(count) : "d"(port) : "memory");
}```

you can steal these

https://github.com/BlueGummi/charmos/blob/main/include/asm.h I have a whole file of inline assembly things just feel free to steal the whole thing lol

[hushed spire]

eyy, this works 😄 thank you!

[hushed spire]

Ah wait no, nvm, still whining about invalid operands lol

Not passing -masm=intel seems to fix it

[hushed spire]

We got serial logging now!

[hushed spire]

Coloring!

[devout yarrow]

very important step

[hushed spire]

I think I'll figure out why my kprintf function doesn't work later, I would like to have a heap allocator sometime this century lol

And I mean I have a printf implementation anyhow, so it's not imperative I get npf working right this second

Guess I gotta figure out how to properly remap the kernel, now that I'm not using my janky bootloader

and why I can't set flags properly

[hushed spire]

Not that interesting, but I can return from a system call now! (Before it just died when trying to return, and I wanted to try an fix that)

[devout yarrow]

we got syscalls before paging

[hushed spire]

Well paging is... kind of implemented, doesn't work properly tho =/

[devout yarrow]

we got userspace before virtual memory

truly the DOS experience

[hushed spire]

Imma take a look at managarms VMM stuff

[devout yarrow]

that not fun

do you wanna look at mine

mine works

[hushed spire]

sure ig

[devout yarrow]

https://github.com/BlueGummi/charmos/blob/main/kernel/mem/vmm.c

i hope it is readable

[hushed spire]

Ah, so you copy the boot cr3 to the kernel's cr3

That's a good idea

imma try that

[hushed spire]

Seems to work! (As usual, please tell me if there's a better way to do it)

ah right

still can't set flags properly for some reason

gotta fix that

huh weird

info tlb seems to report that the W flag is set

but info mem doesn't

[devout yarrow]

this is just me being lazy

hypothetically i could absolutely set up my own page tables and maintain the addresses of all the variables, but do i wanna do allat? nah

you can take a different approach

[hushed spire]

ahh, okay

[devout yarrow]

remember to understand why the code is working

that is the biggest thing for figuring out why it wont be working (which may happen)

[devout yarrow]

are you flushing the tlb 💥

me when i invl my pg

[hushed spire]

writing to CR3 should flush the TLB automatically no?

[devout yarrow]

ya but are you mapping your own thing

im not an expert on all this you should probs read up on it or look at actual good kernel code

[hushed spire]

I'm actually having trouble figuring out where managarms VMM stuff is lol

[devout yarrow]

that is the trouble with managarm

[devout yarrow]

https://github.com/managarm/managarm/blob/aceab70896cd9a71c0f6ebba03b2793191a39840/kernel/thor/arch/x86/paging.cpp

[hushed spire]

Ahh, that makes sense, dunno why I didn't look there

[devout yarrow]

most rational managarm code organization 💥

[sinful igloo]

in managarm the actual page table management code is generic

the architecture-specific code only provides some functions for manipulating ptes and switching to a page table

https://github.com/managarm/managarm/blob/master/kernel/thor/generic/thor-internal/arch-generic/cursor.hpp

and the policy type for x86 https://github.com/managarm/managarm/blob/master/kernel/thor/arch/x86/thor-internal/arch/paging.hpp#L32

[hushed spire]

Oh, good to know, thanks! I'll take a look at it tomorrow!

[hushed spire]

Time to do some more VMM work

Is it weird that I feel like I can read managarm easier than I can read limine lol

Lord I can't figure out why I can't set flags correctly

[hushed spire]

I'm only supposed to set the flags of the last PTE right? (ignoring the Present bit obviously)

for 4KB paging

I've pushed the commit if anyone wanna take a look

[hushed spire]

Also, made a seperate repo for my bootloader https://github.com/BlueSillyDragon/snowboot

[devout yarrow]

dd if=/dev/zero of=snowboot.efi bs=4096 count=0 seek=$$ ( ( ( $$ ( wc -c < snowboot.efi ) + 4095 ) / 4096 ) ) 2>/dev/null``` you have a knack for absurdly cursed hacks for weird shell script issues

patch the script? ❌
whatever the hell this is ✅

honestly

relatable

so real

[hushed spire]

I would patch it but idk why CMake does that 😭

it's really annoying lol

[hushed spire]

okay, imma just work on my PMM some

It inits really slowly

[hushed spire]

I'm going to look at Nyaux for ref because I recently learnt it uses a freelist (and I can't seem to find any OS' with a freelist allocator)

[hushed spire]

Ayy, seems to work! Abysmally slow though (16 seconds to init 💀), so next thing to work on is the whole, have a freelist of memory regions instead of single pages

Function looks much better tho

[devout yarrow]

16 seconds is kinda wild tho

cool tho

me personally, the reason I used a bitmap pmm and literally copied bootloader page tables is because it would take time (at the time) to write something more solid and complex and the duct tape solution worked well because I wanted to get to the more interesting things (but this is not a very good idea)

I'ma go back and fix allat

[devout yarrow]

anyway yeah I decided to go fix it

all done lol

[devout yarrow]

@hushed spire https://github.com/BlueGummi/charmos/blob/main/kernel%2Fmem%2Fvmm.c yo bsd I decided to rewrite my paging to setup my own page tables feel free to look i tried to make it readable 👍

[hushed spire]

Ayy, awesome!

[hushed spire]

Also notice, I'll probably be taking a break from this project, been having confidence issues as of late

Like I feel like I don't really deserve to be working on an OS with how little C++ ik (as silly as that sounds)

And I'm way too harsh on myself for referencing other people's code

So yeah I'll be taking a little break

[devout yarrow]

rewrite it in rust 💥

[hushed spire]

Oh god no XD, I hate how rust looks

[devout yarrow]

what will come first snowOS becoming functional or gta 6

[hushed spire]

probs GTA6

It won't be too long of a break, probably just a week or something

[devout yarrow]

famous last words

have a nice break

[hushed spire]

ty lol

[narrow lake]

a freelist is simpler and faster than a bitmap

[devout yarrow]

oh

ok

👍

[hushed spire]

freelist my beloved ❤️

[hushed spire]

Ayy! I'm back!

Feeling a bit better now

Been reading OSTEP some too (I'm on ch 19 iirc)

First things first, I actually wanna switch to using flanterm instead of my own Terminal emulator (I don't wanna deal with figuring out scrolling rn)

[devout yarrow]

better yet, create networking drivers, port ssh, and you'll never need to write a framebuffer implementation, just connect via ssh

easy peasy

[hushed spire]

Aaand, I'm page faulting, ofc XD

Ah, fixed it!

Needed to pass framebuffer->address instead of just the framebuffer

[hushed spire]

There we go!

[hushed spire]

nanoprintf still not working 😭

[devout yarrow]

2+ years of osdev and still no printf 💥

slow and steady wins the race

[hushed spire]

I have my own implementation of printf, but I wanted to try using npf

[devout yarrow]

what's npf

[narrow lake]

nanoprintf

[devout yarrow]

why is that necessary lol

[narrow lake]

correct, ready made printf impl

[devout yarrow]

understandable

[hushed spire]

Ayy, we're getting there!

[rotund marlin]

so people don't open the nth issue asking why Limine is returning some response as 0 when all it is is them printing wrong >64bit values

yes, it has happened many more times than i can count

[hushed spire]

Now... I'm not supposed print hello world twice, so gotta fix that lol

🎉

[hushed spire]

Got my colored logs back!

And added back serial logging

[visual walrus]

can't you just make it print to flanterm?

oh, i think you got it working

[hushed spire]

Now, slight problem, the PMM takes 4 whole seconds to initialize, that's just unacceptable, so I need to make it, not, do that

[hushed spire]

I just don't know how to actually make a freelist on a per-region basis

[devout yarrow]

define starting addresses in your free list structure and a length of that space

[hushed spire]

Problem I don't get is freeing =/

[devout yarrow]

keep a small array of like length 20 or less of these freelists and just create them when you iterate through the memmap

[devout yarrow]

figure out which of your lists is responsible for the address

each list structure has, say, a start, end and length

check those when you get an address for all your freelists

[hushed spire]

So, if say, address 0x1000 ends up getting freed before 0x2000, do I just make a new freelist entry?

Since then there's a gap

[devout yarrow]

yea

[hushed spire]

Ah okay, that makes sense

[devout yarrow]

hope this helps

[hushed spire]

It does, thanks!

[hushed spire]

Okay that is much better.

[devout yarrow]

you can make limine boot faster

in limine.conf set timeout: 0

see so fast

[hushed spire]

Oh, yeah, good point!

supa fast

[quick hemlock]

you add a field to the freelist entry for how many consecutive pages there. Each region becomes one entry, freeing is easy: each page is its own entry. You dont need to try merge freelist entries, thats a waste.

ugh discord didnt show the rest of the chat, seems you figured it out

[devout yarrow]

lol how does that happen

oh it didnt load it 💥

[hushed spire]

Yeah, although I was remerging the page if it was adjacent to the start of the first entry, but it seems like it's better to create a seperate entry for it regardless

Anyhow, gotta figure out why I can't map pages properly 😭

[hushed spire]

I was being silly lol

It's fixed now

Now gotta make a function to unmap a page

Ayy, it works!

Now that I got that working, I'll build my own page tables instead of copying the bootloader's ones

Just need to figure out how to remap the kernel lol

[devout yarrow]

add a symbol in the linker script to mark the end of the kernel's virtual memory

you can use the limine memmap_request and just map allat

for mapping the kernel i believe there's an executable address request but you can just map starting at 0xffffffff80000000 since that's probably specified in the linker script as the start of the kernel

have fun

[hushed spire]

I figured, but iirc certain parts are supposed to be marked read only, that's mainly what I'm trying to figure out

[hushed spire]

Ah, my PMM isn't quite working correctly

[hushed spire]

Oh, I think it works now

Uh, I feel like this is wrong though

ah fixed it

just a print error lol

used %x instead of %lx

Hmm, that's definitely not correct =/

[devout yarrow]

show linker script + how you are getting those numbers

[devout yarrow]

remember, you want to do

extern uint64_t __kernel_virt_end;```

and access it like

```c
(uint64_t) &__kernel_virt_end;

note the &

[hushed spire]

Ah, that would explain it lol

[devout yarrow]

it's a little strange to wrap your mind around but what that's saying is place the symbol "at the end of the binary", so the value of the symbol is just whatever but the actual address of it lines up with the end of the kernel

[hushed spire]

Okay yeah, that fixed it!

[hushed spire]

Eyyy! There we go!

[finite copper]

u may wanna edit your inital thread message

[finite copper]

good job

[hushed spire]

wdym?

[finite copper]

it says aquaos

[hushed spire]

thankie!

[finite copper]

also btw next would probs be to port uacpi

what build system do u use

[hushed spire]

CMake

[hushed spire]

Wouldn't this go in userspace?

[finite copper]

no

usually the aml shit is in kernel space

at least for a monolithic kernel

linux and windows have this in kernel space, along with most hobby oses

[hushed spire]

microkernel here lol

[finite copper]

@marsh pecan pmos is an outliar

you would probs wanna ask him

i dont know much about how microkernels work

or uh well

any managarm dev lol

[marsh pecan]

Managarm has uACPI in kernel

[finite copper]

pmos doesnt i believe

[marsh pecan]

No

[finite copper]

i guess its a matter of choice

for your microkernel

id say uacpi in the kernel would be easiest and fastest

[hushed spire]

So general consensus seems to be put uACPI in kernel-space

[marsh pecan]

Yeah, ACPI is kinda tied to a bunch of things

[finite copper]

pretty much

it has to do some privledged instructions

[marsh pecan]

It's the straightforward thing to do

[marsh pecan]

Not really

[hushed spire]

wait don't I need I heap allocator first?

[finite copper]

like reading and writing the pciconfig space, reading and writing to ports with inb and outb

[finite copper]

i mean other then those

[marsh pecan]

I've put PCI stuff in userspace as well

[finite copper]

oh

interesting

[hushed spire]

Yeah pretty sure I need a heap allocator first

[finite copper]

i can give you resources on how to implement a slab allocator

if you'd like

:^)

[hushed spire]

yes please!

[marsh pecan]

I just used dlmalloc

[finite copper]

https://people.eecs.berkeley.edu/~kubitron/courses/cs194-24-S14/hand-outs/bonwick_slab.pdf

its quite simply

start of the page has a slab header that store a pointer to another page storing another slab header and a freelist of "objects", basically the page gets divided into objects of equal size

when you want for example an 8 byte allocation

you'd have a slab "cache" of 8 byte slabs (pages with slab headers and objects)

you look for any free objects on the slab header

if not go to the next slab

if there is, pop it off and return it

and freeing is just, getting the start of the page with some funny bitmask magic (making the address you recieved page aligned) and putting the object back on the list

thats a shrimple explaination of it

[devout yarrow]

it's also best to keep a fallback allocator for big allocations since slabs are only better for smaller ones

[finite copper]

cache meaning just a list of slabs

[hushed spire]

Huh, okay, that doesn't seem to horribly complicated

[finite copper]

i.e. your vmm for bigger allocations

[devout yarrow]

so you can add a magic number to the header to tell it whether it's allocated by the slab allocator or some fallback bitmap allocator or whatever

[finite copper]

in nyaux, i just do sized frees

i dont have to store addit metadata

but that works too

[novel briar]

same here

[finite copper]

if you dont already know, your vmm has to allocate virtual addresses and allocate and map them

[hushed spire]

Okay, so I will get started on that! (after a rq break lol)

[finite copper]

ofc, take your time :)

[hushed spire]

So, before I jump into making a Slab Allocator, I should probably first make my VMM... actually, manage virtual memory (since right now all it can really do is page table manipulation)

Which means, time to learn how an AVL Tree works

[marsh pecan]

Read the second part of this paper

[hushed spire]

Oh, okie

Uh, what exactly do you mean by second part 😅

[devout yarrow]

why would this be needed

and how come you don't know how this works anyways

[hushed spire]

🤷‍♂️ leave me alone Imma beginner XD

[devout yarrow]

arent u a gamedev

[hushed spire]

AVL Tree-based VMM

[hushed spire]

yes

[devout yarrow]

gamedev is also like home of the data structures

💥

[hushed spire]

good thing I'm learning them now ig

[devout yarrow]

how come you decided on this

[hushed spire]

Seems like a good way of managing virtual memory

[devout yarrow]

is it the vibes of the data structure

where did u see an AVL tree used to allocate memory

I'm curious

[hushed spire]

https://osdev.wiki/wiki/Page_Frame_Allocation under Virtual Memory Allocator

[devout yarrow]

why not check out slab allocators

[hushed spire]

I'm doing a slab

AVL is specifically for the VMM I'm pretty sure

[devout yarrow]

what do you mean you're doing a slab and you're using an AVL

[hushed spire]

Since I need to know what Virtual Address spaces are used and which ones are not

[hushed spire]

Am I going mad?

VMM is seperate from slab no?

[devout yarrow]

please specify what you mean by "VMM"

virtual memory manager sure

but is it an allocator?

the paging system?

kmem_cache
├── object_size
├── slabs_full → [slab] → [slab] ...
├── slabs_partial → [slab]
├── slabs_free → [slab]
└── ...

slab
├── inuse
├── free_list → obj → obj → ...
├── next
└── data → [obj][obj][obj]...
``` this is how a slab allocator would work

no AVL tree

[hushed spire]

Well from my understanding it's a PMM, but for virtual memory, basically. It keeps track of what virtual addresses are currently in use. So for example, I might have a function called vmmAlloc() which asks the PMM to allocate a page, after it gets that page, it then looks for a VA to map it to, maps it, and returns the address of it

[marsh pecan]

vmem

[hushed spire]

ah okay, thx!

[devout yarrow]

so a slab allocator is a virtual memory allocator

[hushed spire]

I thought it's a heap allocator

[devout yarrow]

i mean if you're funny you can figure out how to use it as a physical memory allocator too

[devout yarrow]

that's what that part of memory is called

all that memory is the "heap"

[marsh pecan]

For physical memory allocator just use freelist

[hushed spire]

I'm confuzzled lol

[devout yarrow]

the "heap" is an OS-defined region of memory that you allocate and manage, the "heap" is essentially that non-stack part of memory that your physical memory manager is keeping track of, and the virtual memory manager/allocator will allocate from/to the heap

basically all that stuff that isn't your kernel or the stack, and isn't in use by any MMIO? that's your heap

alllllll that RAM

usually people don't say "heap" here since "heap" is a bit more of a userland abstraction

but yeah it is one

[marsh pecan]

Idk, you basically need something like mmap or sbrk

For both the userspace and the kernel

The simplest thing for virtual memory is a bump allocator

Which is totally fine in the beginning, the only "issue" is that you can't free memory, which is not a problem in the beginning

[hushed spire]

So what the heck does the OSDev wiki mean when it says this

Linux has used AVL trees for virtual addresses management for quite a while. Note however that it uses it for regions (like what you find in /proc/xxxx/maps), not for a malloc-like interface.

[marsh pecan]

Idk

[devout yarrow]

I believe this is keeping track of address spaces and not allocations

[marsh pecan]

Osdev wiki is full of misinformation

[devout yarrow]

once you get to userspace you need something to create address spaces for all your cute little jobs/tasks/processes

[marsh pecan]

Also this probably refers to userspace memory

[devout yarrow]

i believe that's what that is for

[hushed spire]

ohhh, okay

[marsh pecan]

I also use a BST

In my kernel

For that

[hushed spire]

So is the VMM just a page table manager or something?

[devout yarrow]

no it keeps track of allocations so you can allocate and free memory

[marsh pecan]

The terminology is bad

It could mean several things

[hushed spire]

I'm so confuuuused XD.

[devout yarrow]

i think he's talking about an allocator here

[marsh pecan]

Yes, but it's like 3 allocators stacked

[devout yarrow]

ok ok ok

vmm -> manages virtual memory to physical memory translations, paging system, etc.

virtual memory allocator -> allocate and free virtual memory

use that instead

less confusing

so now, what is confusing about virtual memory allocators?

[marsh pecan]

Also there was a discussion that this was basically invented by osdev forums people

[devout yarrow]

sanity

[hushed spire]

So, I just trust myself not to use already used VAs?

[marsh pecan]

And in big kernels it's one entity

[marsh pecan]

No, you have different allocators for that

Again, the simplest is a bump allocator

[devout yarrow]

I think he wants to spend a lot of time making things correct and complex tho

2 weeks of printf 💥

[hushed spire]

it didn't take me that long 😭

[marsh pecan]

This works just fine. You will know when you would need to replace it

[devout yarrow]

ok so your virtual allocator provides these interfaces

void *kmalloc(size);

and

void kfree(addr/*, optional size*/);

so look into algorithms that let you make that happen

[marsh pecan]

And kmalloc calls mmap

Or sbrk

So mmap or sbrk is virtual memory allocator

[devout yarrow]

yes

[hushed spire]

okay... I think I get it

I mean I'll still learn how an AVL Tree works since I should know regardless lol

[marsh pecan]

There are other binary trees

[devout yarrow]

no datastructure knowledge osdev challenge 💥

[hushed spire]

Binary trees in general, I suppose I should've said

[hushed spire]

Seems like all a bump allocator is, say my heap starts at 0x1000 (for simplicity's sake), and I make an allocation of 8 bytes, I would return 0x1000 to the caller, and then the heap starts at 0x1008

With the downside that I can only free all memory at once

slab doesn't seem horribly hard though, so I'll still probably just stick with that

(I'll make a bump just for the sake of practice tho)

[hushed spire]

ezpz

Now on to mah slab

[finite copper]

what??? no???

[finite copper]

gamer

[hushed spire]

...so I'm not insane?

[finite copper]

how tf is a slab allocator a virtual memory allocator

all it does is DIVIDE pages into smaller allocations

that is it

how you access those pages could be via the hhdm or wtv the hell u do

but its not related to allocating vmem addr

[hushed spire]

okay, I didn't think so

Still not entirely sure if I actually need the whole AVL tree thing in the kernel tho

[devout yarrow]

AVL trees seem to be for managing userspace address spaces between processes

i guess you can try to implement that for allocations

it would be interesting

massive overhead probably

[hushed spire]

this is what I was guessing

[devout yarrow]

i think i have brain damage

I was debugging my xhci initializer code for ten minutes and the bug was that I memzero'd a struct AFTER i set the fields

💥

holy moly

[hushed spire]

[devout yarrow]

genuinely like a fractional IQ score, it works fine now

[hushed spire]

I ended up doing a similar thing when creating the page tables lol

[devout yarrow]

minds great alike think or whatever the saying is...

[hushed spire]

Slab layout for small objects doesn't seem to hard, big objects seem to be a bit more complicated though

[hushed spire]

So I might have somthing like this for a cache

struct Cache {
    uint64_t objSize;
    Slab *slabsEmpty;
    Slab *slabsPartial;
    Slab *slabsComplete;
};

[finite copper]

yea that works

[hushed spire]

😄

[hushed spire]

I have made the most hacky of hacky slab initializations XD

void initSlab(uint64_t hhdm) {
    kprintf(SLAB, "Initializing Slab Allocator...\n");
    caches = reinterpret_cast<SlabCache *>(pmmAlloc() + hhdm);
    caches[0] = createCache(8);
    kprintf(SLAB, "Caches located at 0x%lx\n", caches);
    caches[0].slabsComplete = reinterpret_cast<Slab *>((pmmAlloc() + 0x1000 - sizeof(Slab)) + hhdm);
    kprintf(SLAB, "8 byte cache starting slab header located at 0x%lx\n", caches[0].slabsComplete);
    caches[0].slabsComplete->prev = nullptr;
    caches[0].slabsComplete->next = nullptr;
    caches[0].slabsComplete->head = reinterpret_cast<uint8_t *>((uint64_t)caches[0].slabsComplete & ~0xfff);
    kprintf(SLAB, "First free 8 byte object at 0x%lx\n", caches[0].slabsComplete->head);
    uint64_t next = (uint64_t)caches[0].slabsComplete->head;
    next += caches[0].objSize;
    uint8_t *current = caches[0].slabsComplete->head;
    while (next != (uint64_t)caches[0].slabsComplete) {
        *current = next;
        current = (uint8_t *)next;
        next += caches[0].objSize;
    }
    kprintf(SLAB, "Slab Allocator Initialized!\n");
}

I mean it's a start lol

Hmm, what's the minimum size a slab could alloc?

I would presume 8, because for small objects it seems like you store the freelist in the objects themselves (kind of like a PMM freelist)

And I can't store a 64bit address in a 4byte object

but that also seems kind of silly

allocating 8 bytes for a char would be wasteful

I'm probably just misunderstanding how I'm supposed to store the freelist

[hushed spire]

So turns out, yeah 8 bytes is the minimum (in most cases)

My init function is still a travesty tho 🤣

[hushed spire]

Made a function to create a new slab

Slab *createSlab(uint64_t objSize, Slab *prevSlab, Slab *nextSlab) {
    Slab *newSlab = reinterpret_cast<Slab *>((pmmAlloc() + 0x1000 - sizeof(Slab)) + hhdmOffs);
    newSlab->head = reinterpret_cast<uint64_t *>((uint64_t)newSlab & ~0xfff);
    uint64_t next = (uint64_t)newSlab->head + objSize;
    uint64_t *current = newSlab->head;
    while (next != (uint64_t)newSlab) {
        *current = next;
        current = (uint64_t *)next;
        next += objSize;
    }
    newSlab->prev = prevSlab;
    newSlab->next = nextSlab;
    return newSlab;
}

Got some basic allocation going!

[hushed spire]

Got a basic kfree now!

[devout yarrow]

stress test when

[hushed spire]

Uh, I dunno how to stress test it lol (It would probably break tbh XD, it's pretty janky)

Also made a funny mistake here lol

while (next != (uint64_t)newSlab) {
        *current = next;
        current = (uint64_t *)next;
        next += objSize;
    }

should've been

while (next < (uint64_t)newSlab) {
        *current = next;
        current = (uint64_t *)next;
        next += objSize;
    }

[devout yarrow]

keep allocating and writing and reading from said allocations until something bad happens

like run a bunch of loops

[hushed spire]

Ahh, okay

[devout yarrow]

2 million 8 byte allocations, memset and then memcpy and memcmp them with a buffer, then do 1 million 16 byte allocs, etc

probably less than a million but u get the jist

[hushed spire]

Also use a for loop now for initializing the cache

void initSlab(uint64_t hhdm) {
    kprintf(SLAB, "Initializing Slab Allocator...\n");
    hhdmOffs = hhdm;
    caches = reinterpret_cast<SlabCache *>(pmmAlloc() + hhdm);
    memset(caches, 0, 0x1000);

    uint64_t objSize = 8;
    for (uint64_t i = 0; objSize <= 256; i++) {
        caches[i] = createCache(objSize);
        objSize *= 2;
    }

    for (uint64_t i = 0; caches[i].objSize != 0; i++) {
        caches[i].slabsEmpty = createSlab(caches[i].objSize, nullptr, nullptr);
    }

    kprintf(SLAB, "Caches located at 0x%lx\n", caches);
    kprintf(SLAB, "Slab Allocator Initialized!\n");
}

[hushed spire]

Also it breaks fairly quickly under a stress test, no surprise there lol, stuff is only half implemented so far

Head now gets properly set to a nullptr, and I can catch this instead of just faulting

Gotta make it get the next slab instead of just whining that it's out of memory tho

[devout yarrow]

💥

[hushed spire]

Can get the number of slabs in a cache (and return the last of those slabs)

[hushed spire]

Okay, allocating 4096 uint64_ts seems to work

That's a good sign

My Slab doesn't crash and burn 🎉

Still unfinished

but it's good that it doesn't just immediately break lol

now I need to actually manage what slabs are in what lists (since rn all slabs are presumed to be in the Empty list)

[hushed spire]

Hmm, would I need some way to know the size of the object being freed? Since, if say, you free the last object on a slab, then that slab needs to be moved the the empty list, and for that I need to know which cache that slab belongs to

[hushed spire]

Now I create a new cache when an allocation is requested of a size which doesn't have a cache

[devout yarrow]

you can literally

put a [size] right before the ptr returned

and on frees

subtract from that ptr you are given to retrieve the size

basically keep a little piece of metadata around the alloc regarding its size

[hushed spire]

okie, thanks!

tomorrow we cooking

🔥

[hushed spire]

wait this math seems off =/

[devout yarrow]

whoops typo

whatever lol

basically put a small size thing before the pointer you return

and you can subtract the size of that integer or whatnot to retrieve the alloc size

just like never ever ever overwrite that when you alloc

because big bad happens

[hushed spire]

...this still seems off, doesn't alloc_size = ptr - sizeof(u64) actually mean alloc_size = address - 64?

cuz, pointer arithmetic

[devout yarrow]

u64 isn't 64 bytes I think

[hushed spire]

u64 is 8 bytes

which means sizeof returns 8

if ptr is a u64*

doesn't this mean the address is addr - 64

because arithmetic operations on a pointer work in the pointer's type size?

[devout yarrow]

you can cast it to a byte pointer

anyways my brain is a little fried but u get the idea

[hushed spire]

yeah I get the idea lol

[devout yarrow]

basically just have a little field before the allocated area

[devout yarrow]

just never ever overwrite that because I was being silly and did alignment wrong and overwrote that

also a cool thing you should do is implement a fallback allocator

so if the allocation is large you use that instead of a slab

like a bitmap or whatnot

slabs are best for allocations under one page

and you can use a magic number right before the pointer the allocator returns to indicate it came from the bitmap

have fun this stuff is hopefully not too hard

[hushed spire]

hasn't been too bad so far!

[hushed spire]

Alright, time to work on freeing and actually putting slabs in the right list

[hushed spire]

okay storing metadata only affects how many 8 & 9 byte objects that can be stored in a slab, doesn't matter for bigger slabs

[hushed spire]

oop, I'm GPFing lol

Yeah that's definitely not correct

[hushed spire]

Oh, think I might know the issue now

nvm, no clue lol

[hushed spire]

Ah, ubsan is complaining about misaligned pointers again

tinyubsan: use of misaligned pointer at file /home/bluesillydragon/SnowOS/yuki/kernel.cpp, line 161, column 9
tinyubsan: use of misaligned pointer at file /home/bluesillydragon/SnowOS/yuki/kernel.cpp, line 162, column 66

which is probably true but I dunno how to avoid that with metadata

Unless I'm going to make the metadata take up 8 bytes instead of 2

I might just do sized frees for now

[wise ridge]

i mean, use aligned reads?

for example by reading with __builtin_memcpy_inline

[hushed spire]

uh, idk what that is 🤣

I think I'll just do sized frees for now, I can always come back to it later

And literally all it's used for is figuring out which cache a slab belongs to (at least that's what I need it for)

Gonna change my getLastSlab() function to getSlab() no clue why I'm getting the last slab in a cache instead of the first one

[hushed spire]

Now I'm freezing for some reason 😭

Working slightly better now but I'm still freezing

Mmm, don't you love nested ifs?

if (caches[i].slabsPartial != currentSlab) {
        currentSlab->next = caches[i].slabsPartial;
        caches[i].slabsPartial = currentSlab;

        if (caches[i].slabsEmpty == currentSlab) {
            caches[i].slabsEmpty = currentSlab->next;
        }
    }

I'll probably make a function to find a slab within a certain list

I can't wait to look back at this stuff in like, a few months to a year later, and say "Wow, that was really stupid" XD

[hushed spire]

Alright, slabs get properly moved to the partial list now (well maybe not "properly" someone would probably has an aneurism if they had to read my code lol)

I need a break now

[hushed spire]

Aight, I'm back

Lemme work on freeing stuff

I feel like I'm doing some of these things in the most ineffecient way possible

[hushed spire]

This is such heresy lol

if (slab->refCount == 0) {
        slab->next = caches[i].slabsEmpty;
        caches[i].slabsEmpty = slab;
    } else if (findSlab(caches[i], slab, FULL) == slab) {
        // When it's on the top
        if (slab->prev == nullptr) {
            slab->next->prev = nullptr;
            caches[i].slabsFull = slab->next;
        } else {
            slab->prev->next = slab->next;
            slab->next->prev = slab->prev;
        }

    }

https://github.com/BlueSillyDragon/SnowOS/blob/main/yuki/mm/slab.cpp

Any advice would be greatly appreciated

Now I'm going to think of a way to make this not suck

[hushed spire]

Made a little list of things to work on with the slab

Issues with SnowOS' Slab

1. Overall just very hacky (Fix this before anything else)
2. Searches all caches for n sized object every time an allocation is made, store a the index of the most recently accessed cache, and check this when allocating to save some time
3. We do not support Large Slabs atm, add support for this

So tomorrow imma work on this

uACPI stuff soon, hopefully

I'll need to check what other things uACPI needs

[finite copper]

theres no such thing as large slabs

or well

there is for some slab implementations

but that sounds like your overcomplicating things for things you dont need to worry about

your slab just divide page allocations

thats all it should do rlly

[hushed spire]

Ah

[finite copper]

as for number 2, it doesnt matter your gonna have to search to find the biggest cache

its just a quick for loop

usually u have 7 caches so its FINE

[hushed spire]

True, just random thing I thought of lol (seemed like it was a bit unnecessary to search the caches every time an allocation was made)

[finite copper]

i mean sure your os lol

do what u want

im just gonna eat this chocolate while i struggle with the fucking ps2 controller for some reason

[hushed spire]

good luck!

[finite copper]

im not having lukc

luck*

but thanks

[hushed spire]

Saw everyone talking about doom in Nyaux thread lol, but it kind of got me thinking, how hard would porting Cave Story to my OS be? 🤔

Probably very difficult

Totally possible though

Cave Story has like, 5 million ports XD

Ah, this seems like what everyone uses https://github.com/EXL/NXEngine

Welp, adding that on the list of things to do when I get userspace up and running

[hushed spire]

Aight, time to actually think of a better way to implement stuff lol

Let's see how long this takes me

I suppose the first suspect thing might be this?

Slab *getSlab(SlabCache cache) {
    if (cache.slabsPartial == nullptr) {
        if (cache.slabsEmpty == nullptr) {
            cache.slabsEmpty = createSlab(cache.objSize, nullptr, nullptr);
        }
        return cache.slabsEmpty;
    }
    return cache.slabsPartial;
}

cuz nested ifs

I mean idk

Doesn't look that bad to me

Not entirely sure if this function is needed

Slab *findSlab(SlabCache cache, Slab *slabToFind, SLAB_TYPE type)

I mean I feel like the main culprits are stuff like thisZ

if (slab->refCount == 0) {
        kprintf(SLAB, "Freed last object of slab, moving to empty list...\n");
        slab->next = caches[i].slabsEmpty;
        caches[i].slabsEmpty = slab;
    } else if (findSlab(caches[i], slab, FULL) == slab) {
        kprintf(SLAB, "Moving to partial list\n");
        // When it's on the top
        if (slab->prev == nullptr) {
            slab->next->prev = nullptr;
            caches[i].slabsFull = slab->next;
        } else {
            slab->prev->next = slab->next;
            slab->next->prev = slab->prev;
        }
        slab->next = caches[i].slabsPartial;
        caches[i].slabsPartial = slab;
    }

What even is this 😭

This doesn't even work

if (slab->refCount == 0) {
        kprintf(SLAB, "Freed last object of slab, moving to empty list...\n");
        slab->next = caches[i].slabsEmpty;
        caches[i].slabsEmpty = slab;
    }

Move it to the empty list

But I don't move it off the partial list

[hushed spire]

Okay this isn't quite as silly as I thought it was

I have to traverse a linked list in order to find an object

Dunno why I thought there was any other way

This should work properly

if (slab->refCount == 0) {
        kprintf(SLAB, "Freed last object of slab, moving to empty list...\n");

        slab->next = caches[i].slabsEmpty;
        caches[i].slabsEmpty = slab;

        // Remove off the partial list (this'll break if freeing from a free slab, but just, don't do that)
        caches[i].slabsPartial = slab->next;

        slab->prev = nullptr;

    }

wait

no yeah

that's right

Now this I can just say != nullptr instead of == slab

else if (findSlab(caches[i], slab, FULL) == slab)

I mean it's probably pretty much the same, but I like how that looks better

This looks suspect

slab->next->prev = nullptr;

I don't check if there even is a next

so if next is a nullptr, I'm writing to a nullptr, and that's a big no no

I'm realizing this is alot of copying of code lol

Slab *findSlab(SlabCache cache, Slab *slabToFind, SLAB_TYPE type) {
    Slab *currentSlab =  nullptr;
    switch (type) {
        case EMPTY:
            currentSlab = cache.slabsEmpty;
            while (currentSlab != nullptr) {
                if (currentSlab == slabToFind) {
                    return currentSlab;
                }
                currentSlab = currentSlab->next;
            }
            break;
        case PARTIAL:
            currentSlab = cache.slabsPartial;
            while (currentSlab != nullptr) {
                if (currentSlab == slabToFind) {
                    return currentSlab;
                }
                currentSlab = currentSlab->next;
            }
            break;
        case FULL:
            currentSlab = cache.slabsFull;
            while (currentSlab != nullptr) {
                if (currentSlab == slabToFind) {
                    return currentSlab;
                }
                currentSlab = currentSlab->next;
            }
            break;
    }
    return nullptr;
}

Alright that's better

    Slab *currentSlab =  nullptr;
    switch (type) {
        case EMPTY:
            currentSlab = cache.slabsEmpty;
            break;
        case PARTIAL:
            currentSlab = cache.slabsPartial;
            break;
        case FULL:
            currentSlab = cache.slabsFull;
            break;
    }
    while (currentSlab != nullptr) {
            if (currentSlab == slabToFind) {
                return currentSlab;
            }
            currentSlab = currentSlab->next;
    }
    return nullptr;

Ik we don't like nested ifs but I didn't know what else to do

caches[i].slabsFull = slab->next;
            if (slab->next != nullptr) {
                slab->next->prev = nullptr;
            }

I don't write to nullptrs now tho

so that's good

Did this for a stress test and... nothing super bad seemed to happen.

uint64_t *buf = (uint64_t *)kmalloc(sizeof(uint64_t));
    memset(buf, 'S', sizeof(uint64_t));

    for (uint64_t i = 0; i < 1000000; i++) {
        uint64_t *test = (uint64_t *)kmalloc(sizeof(uint64_t));
        memset(test, 'S', sizeof(uint64_t));
        int rv = memcmp(test, buf, sizeof(uint64_t));
    }

🎉

I was honestly expecting a slab implementation to be harder than it actually was

Now I gotta see what else uACPI needs

We gonna cook 🔥

[devout yarrow]

💥

[devout yarrow]

it needs some scheduler things for the fancy functionality 👀

[hushed spire]

I don't need fancy functionallity atm

[devout yarrow]

real and true...

[hushed spire]

Okay, so uACPI needs someway to get the RSDP, that makes sense

Lemme write that, shouldn't be too hard

Yep, really simple

Uh, does uACPI need the physical address of the RSDP or the virtual address?

Oh wait

It says it takes the physical address

Ah, I might need my whole AVL Tree thing

void *uacpi_kernel_map(uacpi_phys_addr addr, uacpi_size len);

This doesn't tell me what VA it wants the PA mapped to

So I need to know what address to map it to

[devout yarrow]

limine directly provides this to you lol

__attribute__((
    used,
    section(".limine_requests"))) static volatile struct limine_rsdp_request
    rsdp_request = {.id = LIMINE_RSDP_REQUEST, .revision = 2};```

it truly is that shrimple

rsdp_request.response->address

💥

[devout yarrow]

ur supposed to return it

void *vmm_map_phys(uint64_t addr, uint64_t len) {

    uintptr_t phys_start = PAGE_ALIGN_DOWN(addr);
    uintptr_t offset = addr - phys_start;

    uint64_t total_len = len + offset;
    uint64_t total_pages = (total_len + PAGE_SIZE - 1) / PAGE_SIZE;

    if (vmm_map_top + total_pages * PAGE_SIZE > VMM_MAP_LIMIT) {
        return NULL;
    }

    uintptr_t virt_start = vmm_map_top;
    vmm_map_top += total_pages * PAGE_SIZE;

    for (uint64_t i = 0; i < total_pages; i++) {
        vmm_map_page(virt_start + i * PAGE_SIZE, phys_start + i * PAGE_SIZE,
                     PAGING_PRESENT | PAGING_WRITE);
    }

    return (void *) (virt_start + offset);
}``` something like this

just have some starting map and map from there

#define UACPI_MAP_BASE 0xFFFFA00000000000
#define UACPI_MAP_LIMIT 0xFFFFA00000100000``` you can define a region to just steal addresses from

[hushed spire]

Okay yeah, good to know, thanks!

[hushed spire]

Ayy!

ah, wait a sec

slight issue here lol

Aight, fixed it!

[devout yarrow]

a cool little addition could be a very simple aligned allocator wrapper that just aligns allocations, it'll be helpful in the future

one sec I can give u one if u want

void *kmalloc_aligned(uint64_t size, uint64_t align) {
    uintptr_t raw = (uintptr_t) kmalloc(size + align - 1 + sizeof(uintptr_t));
    if (!raw)
        return NULL;

    uintptr_t aligned = (raw + sizeof(uintptr_t) + align - 1) & ~(align - 1);
    ((uintptr_t *) aligned)[-1] =
        raw; 

    return (void *) aligned;
}

void kfree_aligned(void *ptr) {
    if (!ptr)
        return;
    uintptr_t raw = ((uintptr_t *) ptr)[-1];
    kfree((void *) raw);
}

just lets you align things to byte boundaries

pretty useful sometimes for some drivers and obvs for things like TSS and whatnot

[hushed spire]

Interestingly, flanterm doesn't wanna print on real hw, idk what I did to make that happen but, okay then

Ah, nvm

It was me doing a silly lol

Works fine now

[hushed spire]

Ah yes, I should probably give this a read https://www.microsoft.com/en-us/research/wp-content/uploads/2019/04/fork-hotos19.pdf

[hushed spire]

I keep having to remind myself that this doesn't mean 0x...2000 is mapped lol

Anyhow!

void *uacpi_kernel_map(uacpi_phys_addr addr, uacpi_size len);
``` and
```c++
void uacpi_kernel_unmap(void *addr, uacpi_size len);
``` are now implemented!

A bit scuffed, but implemeted!

Now we gonna integrate uACPI!

[hushed spire]

Uh, do I do something like

#define UACPI_BAREBONES_MODE

ig I'll try it

Ah, no

[devout yarrow]

compiler flag

-DUACPI_BAREBONES_MODE=true or whatever

[hushed spire]

[hushed spire]

ah okay

excuse me?

clang what're ya doing XD

oh wait

I can't use uacpi_initialize in barebones mode

Ah, I'm supposed to call uacpi_setup_early_table_access()

Ooh, yeah I have definitely done something wrong.

Ah hah! I fixed it!

kinda hackily tho

I really should make my VMM properly manage VAs

[hushed spire]

Also I need to make my kernel panic not completely useless lol

[visual walrus]

btw @hushed spire early table init is good for loading apic and hpet + other stuff

but not peripherals

for that you need the complete one i think

(pci i am sure)

[hushed spire]

Ah, okay

[hushed spire]

Well, I'm a microkernel so I don't think I actually have to talk to PCI devices

[devout yarrow]

you're gonna have to worry about how to provide interfaces for the drivers to do that tho

also these drivers ain't gonna write themselves

[hushed spire]

true true.

[hushed spire]

gimme a sec 😭 I need a scheduler lol

And a TSS

[devout yarrow]

take ur time, rushing it makes things worse usually

I mean on the topic of schedulers, a good starting point is a basic uniprocessor round robin and then you can build up from there

[hushed spire]

okie, good to know, thx!

[visual walrus]

mine likes to corrupt queues for some reason

[hushed spire]

I'm gonna try to work on enabling the LAPIC timer, so hopefully I get to work on scheduling soon

[devout yarrow]

that's a skill issue pal

[visual walrus]

yeah ik

need to find why

[hushed spire]

Alrighty, let's see if I can figure out how to enable the LAPIC

[devout yarrow]

it's super duper trivial you got this

you just map an address that you read from an msr and poke a few things and ur done

[hushed spire]

Ah, that doesn't seem too bad

First I need to figure out how to actually get the APIC from uACPI

[devout yarrow]

for lapic?

no need

[hushed spire]

oh I don't?

[devout yarrow]

#define IA32_APIC_BASE_MSR 0x1B
#define IA32_APIC_BASE_MASK 0xFFFFF000UL

uint64_t apic_base_msr = rdmsr(IA32_APIC_BASE_MSR);
uintptr_t lapic_phys = apic_base_msr & IA32_APIC_BASE_MASK;
lapic = vmm_map_phys(lapic_phys, 4096);```

it's that shrimple

static inline void wrmsr(uint32_t msr, uint64_t value) {
    uint32_t lo = value & 0xFFFFFFFF;
    uint32_t hi = value >> 32;
    asm volatile("wrmsr" ::"c"(msr), "a"(lo), "d"(hi) : "memory");
}

static inline uint64_t rdmsr(uint32_t msr) {
    uint64_t lo, hi;
    asm volatile("rdmsr" : "=a"(lo), "=d"(hi) : "c"(msr));
    return (hi << 32U) | lo;
}```

[hushed spire]

oh, okie

[hushed spire]

I presume Table 12-1. Local APIC Register Address Map in the intel sdm is what I should be looking at

[devout yarrow]

yes I think so

[earnest birch]

Btw as you seem to be doing everything really thoroughly (cudos to you about that btw 🤩). Take a look at x2apic

[visual walrus]

limine gives you lapics

i use uacpi for ioapic

[hushed spire]

Another day, more trying to setup the lapic

It seems like the LVT Timer Register and Spurious Interrupt Vector Register hold the interrupt vectors for the timer int and spurious int

[visual walrus]

if you want on my github i have code for it

it is pretty easy to understand

[hushed spire]

sure

[visual walrus]

YESS, my scheduler works too now

[hushed spire]

yay!

[visual walrus]

i did not add an if

that's why it was failing

lol

btw, here's the code: https://github.com/Diamantino-Op/HorizonOS/blob/main/src/kernel/kernel-x86_64/src/hal/Apic.cpp

i get the lapic id from limine

but it really is not used

you don't need it for init

just run the init code on the core to init

[hushed spire]

okie, I shall give it a read!

[visual walrus]

i still need to add a function to stop the scheduled one-shot timer

but to do that just write 0 to LVT i think

[hushed spire]

Okay, for some reason info lapic isn't reporting the spurious register having changed even tho I'm modifying it, is that normal or...?

mmioOut(reinterpret_cast<uint32_t *>(lapic + LAPIC_SPURIOUS_IVT_REG), (1 << 8) | 0xFF);

[visual walrus]

are you using x2apic or the older version?

bc you access them 2 diff ways

[hushed spire]

ah, okay that might be the issue

[visual walrus]

look at my code

[hushed spire]

idk tbh lol

[visual walrus]

you see that if i enable x2apic, i do not use mmio, but msrs

[hushed spire]

Ahh, okay

[visual walrus]

mc that's how you use x2apics

i also have the function to convert from older apic to x2apic address

[devout yarrow]

you might wanna consider using macros here and defining ANSI constants so this doesnt get out of hand super quickly

and a function to print to both serial + flanterm

also uhm

sizeof != strlen

this will just report the size of char*

that might be why it wasnt printing or something

[hushed spire]

Was taking a little breather, but I'm back now.

[hushed spire]

Huh, really?

[devout yarrow]

yep

[hushed spire]

I mean I do have strlen so idk why I'm not using that

[devout yarrow]

💥

[devout yarrow]

if they were the same then strlen would just use sizeof lol

but one is compile time and the other is runtime

sizeof would report 8 here on x64 since that's the address width 💥

[sinful igloo]

the type of "asdf" is const char[5] not const char *

it decays into a pointer but for sizeof it gives you the amount of bytes in the string

[devout yarrow]

oh yea array decay my beloved

didnt know that applied here too

neat 👍

[hushed spire]

I mean I think either works, so I might as well use strlen since I have it

[devout yarrow]

i guess sizeof would be faster by the most unnoticeable margin since it's compile time

[sinful igloo]

well sizeof is still not != strlen since it counts the null terminator

[hushed spire]

Anyhow, how do I know if I have the old APIC or x2APIC? Do I check the Local APIC Version Register?

[sinful igloo]

since it is an array element

[hushed spire]

Ah okay, it seems like it justs tells me whether or not it's integrated

Ah, okay I just have to enable it myself

I said 'Ah, okay' twice

Lemme write a function for getting stuff via CPUID

[hushed spire]

Ay, it works!

Now I can check if I can use x2ACPI

[hushed spire]

Ayy!

ALRIGHT! 🎉

I somehow disabled APIC tho

I don't think I'm supposed to do that XD

[visual walrus]

lol

[hushed spire]

Aight, fixed it

[visual walrus]

gg

[hushed spire]

Now I need to actually setup the interval

[visual walrus]

it's not that hard

gl

well, i think it's time for me to work on IPC

when i find good resources

so i can make pci drivers

[hushed spire]

good luck!

[visual walrus]

thx

[hushed spire]

Why am I sorely tempted to draw an anime mascot char for my OS 🤣

Anyhow

Seems I use the PIT to calibrate the timer

Soo, time to figure out how to use the PIT

can't be too hard

[visual walrus]

PIt just works with interrupts

and is unreliable bc of that

[hushed spire]

Ah, okay then

I suppose I can use the HPET instead

i think

can you use the HPET to calibrate the LAPIC?

[visual walrus]

yes

[hushed spire]

Okay, that it is then

[visual walrus]

look at my calibrate function

first it tries to use a cpuid

if it cannot use it, it uses the best timer it has

it can be hpet, acpipm, kvm, tsc

pit is the worst one tho

kvm is the best one

[hushed spire]

alr, yeah I figured, I just didn't know if I could use the hpet

[hushed spire]

uh, wuts this one?

[visual walrus]

basically kvm exposes a timer

and it's the most accurate one, being that it's yout hardware one

lol

[hushed spire]

ah okay, that makes sense

I presume it doesn't work on real hw tho

[visual walrus]

yeah nope