FrostyOS | Operating System Development | Page 3

lost schooner Jun 28, 2024, 5:40 AM

#

and serial devices can either be backed by a UART (like on x86 machines), or by linux read/write/lseek syscalls for userland mode

#

I don't know if I have explicitly said this, but I'm not supporting running FrostyOS natively on anything other than linux

#

its too much effort

#

2000th message

lost schooner Jun 28, 2024, 6:10 AM

#

I've implemented backends for VGA output, port 0xE9 debug (which I call BochsDebug), and keyboard input

#

it was very simple

#

mostly just moving code around

#

github copilot helped a lot

#

now that stuff is handled correctly at a TTY level, I need to cleanup file descriptors so they do things properly

dusky orbit Jun 28, 2024, 7:22 AM

#

nice stuff, a little abstraction in the right places helps a lot

lost schooner Jun 29, 2024, 6:18 AM

#

I though I had implemented things correctly until I got a triple fault immediately on boot

#

its a page fault before I even load the GDT...

#

its a write to non-present page

#

oh no

#

looks like a stack overflow

#

a stack trace with 1244 entries is not a good sign

#

it started with a file descriptor failing to be reserved for stddebug

#

and then when that tries to be printed, ubsan catches that the FileDescriptor for stddebug doesn't exist

#

that tries to get printed, which can't be

#

and that continues until a stack overflow happens

#

its so stupid of a bug

#

when I copied a line, I forgot to update 1 of the arguments

#

maybe I should have just read my code before saying something about this triple fault

lost schooner Jun 29, 2024, 6:47 AM

#

the new system works well

lost schooner Jun 29, 2024, 11:41 PM

#

I've been thinking about doing a big refactor of the VMM

#

It's very messy

#

Most of the code is from 1.5 years ago, and my C knowledge wasn't ideal then

#

While I'm at it, I might make some changes to the PMM

#

I currently use a bitmap, but I've been thinking about trying to make a buddy allocator

#

As for the virtual memory allocation, I use dual AVL-trees

#

One for used/reserved pages, sorted by address, with the extra data held at each node being the size up to 63-bits, and the highest bit being 1 for reserved, 0 for used.

#

The other tree is for free pages, sorted by size. The extra data at each node points to the beginning of a dually linked list of addresses.

#

All that is for each instance of the VMM

#

There is an instance for kernel space, which starts at the end of non-canonical address space, and ends at the end of virtual memory.

#

Then each process has their own instance which ranges from 0x1000 to the beginning of non-canonical address space

#

The code that links the PMM and VMM together is strange

#

I have a PageManager class, which follows the same instance stuff as the core VMM

#

It has a standard linked list of allocated objects

#

That part needs optimising

#

I haven't done any proper testing, but I don't think it is fast

#

I probably should be using something like an AVL tree sorted by address for that

lost schooner Jun 30, 2024, 4:06 AM

#

I tried making that change, but it didn't go well

#

I realised that I need to quickly find the PageObject that contains a virtual memory range, not just the base address of the object

#

I might actually need to rewrite the PageManager completely

lost schooner Jun 30, 2024, 4:37 AM

#

I've been thinking about how I want to do this properly

#

I have a function for finding a node with a key equal or greater than the requested key, which I could probably use with some minor modifications

lost schooner Jul 1, 2024, 2:56 AM

#

I just wrote this I already can barely understand it:

uint64_t end = (uint64_t)region.GetEnd();

using namespace AVLTree;
Node const* root = m_allocated_objects.GetRoot();
Node const* current = root;
Node const* lastVisited = nullptr;
Node const* next = nullptr;

while (current != nullptr) {
    if (lastVisited == current->parent) {
        // Coming down to current from parent
        if (current->left != nullptr)
            next = current->left;
        else if (current->right != nullptr) {
            if (region.GetStart() >= ((PageObject*)current->extraData)->virtual_address
                && end <= ((uint64_t)(((PageObject*)current->extraData)->virtual_address) + ((PageObject*)current->extraData)->page_count * PAGE_SIZE))
                return (PageObject*)current->extraData;
            next = current->right;
        }
        else {
            if (region.GetStart() >= ((PageObject*)current->extraData)->virtual_address
            && end <= ((uint64_t)(((PageObject*)current->extraData)->virtual_address) + ((PageObject*)current->extraData)->page_count * PAGE_SIZE))
                return (PageObject*)current->extraData;
            next = current->parent;
        }
    }
    else if (lastVisited == current->left && current->right != nullptr) {
        // Coming up to current from left child, and there's a right child
        if (region.GetStart() >= ((PageObject*)current->extraData)->virtual_address
        && end <= ((uint64_t)(((PageObject*)current->extraData)->virtual_address) + ((PageObject*)current->extraData)->page_count * PAGE_SIZE))
            return (PageObject*)current->extraData;
        next = current->right;
    }
    else
        next = current->parent; // Coming up to current from right child or there's no right child

    lastVisited = current;
    current = next;
}

return nullptr;

#

I hope it works

#

its still a while before I can test it

dusky orbit Jul 1, 2024, 3:41 AM

#

bloody hell lol

#

at least its lot of the same calculations, you could reduce that into a few variables that get calculated at the start of each iteration

lost schooner Jul 1, 2024, 4:05 AM

#

I was going to go for a recursive approach, but I decided I didn't want to fill the stack

lost schooner Jul 1, 2024, 5:18 AM

#

I think I might finally have finished re-writing the page manager

#

i have very strong suspicions that it won't work first try

#

but I guess I should stop procrastinating and test it...

#

this is fun:

#

it says no VGA device, but it does exist

#

the ubsan violation is just a nullptr dereference. its a combination of a use after free and me not checking if something is null before writing to it

#

next bug:

lost schooner Jul 1, 2024, 5:45 AM

#

that is in memory which should be allocated for the heap

#

the PageObjects in the PageManager are fine

#

the memset in this page faults, leaving no decent stack trace:

size_t numPages = DIV_ROUNDUP((size + sizeof(BlockHeader)), PAGE_SIZE);
size_t numBytes = numPages * PAGE_SIZE;
void* ptr = g_KPM->AllocatePages(numPages);
memset(ptr, 0, numBytes);

#

i wonder if I'm some how trashing the page tables

#

unlikely, but possible

#

this is bad:

#

just had to restart GDB

#

anyway, the page object is correct, yet the pages aren't being mapped properly

#

somehow 10 AVL tree nodes have been allocated, when really, only 3 have been

lost schooner Jul 1, 2024, 7:32 AM

#

That last message was a bit unclear

#

I have a pool of AVL tree nodes inside the kernel to use when the heap is not operational, such as during VMM start-up

#

The number of allocated nodes is 10

#

But I can only see 3 actually being used

#

That means there is a memory leak somewhere, potentially in the virtual address space allocator

lost schooner Jul 1, 2024, 8:05 AM

#

ohh

#

i know why

#

I forgot about the global instance

#

not just the kernel specific instance

#

I guess I have to go back to hunting for things that look unusual

#

it still looks like regions aren't being mapped properly

#

there are 2 main regions which are allocated this early in boot, one for the eternal heap, and one for the normal heap

#

the PageObjects in the PageManager say that these range from:

0xffffffff8020d000-0xffffffff8040d000
0xffffffff8040d000-0xffffffff8080e000

#

but qemu says otherwise

#

#

i think i found it

#

this tiny expression:

(void*)((uint64_t)virt_addr + i * count)

#

should be:

(void*)((uint64_t)virt_addr + i * PAGE_SIZE)

#

got to uACPI init this time before failing

#

KERNEL PANIC!
UBSan: type mismatch at /home/frosty515/dev/FrostyOS/FrostyOS/kernel/src/Data-structures/LinkedList.cpp:160:20
 on CPU 0
RAX=000000000000006f  RBX=ffffffff801e76d8  RCX=ffffffff801e767c  RDX=ffffffff800500e9
RSI=000000000000000a  RDI=ffffffff801e76d8  RSP=ffffffff801e7678  RBP=ffffffff801e76a0
R8 =ffffffff8020af00  R9 =0000000000000000  R10=0000000000000003  R11=0000000000000000
R12=ffffffff8040d600  R13=000000000000000c  R14=ffffffff8040d6e0  R15=0000000000000003
RIP=ffffffff801033d0  RFL=0000000000000282
CS=0008  DS=0010
Stack trace:
ffffffff801033d0
ffffffff8010344b
ffffffff800d19a7
ffffffff8002622c
ffffffff8002b482
ffffffff80033476
ffffffff80019575
ffffffff8005fba1
ffffffff80066984
ffffffff80016e23
ffffffff8005a7f5
ffffffff8010028a
ffffffff8005e8a6
ffffffff8000ed3b

#

that is one big stack trace

lost schooner Jul 1, 2024, 8:59 AM

#

looks like the virtual address space allocator's structures are getting damaged

#

i see part of the problem

#

the parent pointer in the AVL tree nodes isn't being set properly

#

it is set fine before balancing, but then isn't after that

lost schooner Jul 1, 2024, 9:51 AM

#

alright, I'm abandoning the idea of AVL tree nodes having a parent pointer. I barely understand AVL trees, and all this stuff isn't helping

#

it might make other things more complex, but at this point I don't really care

hollow trail Jul 1, 2024, 10:18 AM

#

couldnt you just test it in userspace

#

where u can at least debug it properly

lost schooner Jul 1, 2024, 10:20 AM

#

Maybe

#

Ill try that tomorrow

#

If I try do any more today, I'll just make things worse

lost schooner Jul 2, 2024, 12:14 AM

#

for anyone that is curious, this is the current loc:

lost schooner Jul 2, 2024, 12:40 AM

#

lost schooner ``` KERNEL PANIC! UBSan: type mismatch at /home/frosty515/dev/FrostyOS/FrostyOS/...

after disabling a bunch of stuff, I discovered that this ^^^ is due to my MapPages and UnmapPages functions being bad

#

well this is odd:

(gdb) p *(LinkedList::Node*)($10.extraData)
$11 = {previous = 0xffffffff8163b000, data = 3, next = 0xc}
(gdb) p *($11.previous)
$12 = {previous = 0x20b454445344, data = 2329284886210418433,
  next = 0x2020202043505842}

#

the only node in the AVL tree for free pages in the virtual address space allocator contains a bogus linked list

#

but if I read it as a PageObject, I got some results:

(gdb) p *(PageObject*)($10.extraData)
$14 = {virtual_address = 0xffffffff8163b000, page_count = 3,
  flags = 12, perms = PagePermissions::READ_WRITE}

#

QEMU says that region is mapped:

#

so that AVL tree node is being overwritten by the PageManager

#

so my PageManager::FindObject function is most likely the source of the issue

#

maybe a use after free somewhere

lost schooner Jul 2, 2024, 1:14 AM

#

something is very wrong

#

the page manager says there are 17 allocated regions, but there are actually only 9 in its tree

#

I wonder how hard it would be to get some kind of KASAN working

hardy lake Jul 2, 2024, 1:16 AM

#

lost schooner I wonder how hard it would be to get some kind of KASAN working

well...

#

depends on how you do it

#

if you do it the way the ASAN developers did

#

not that hard

#

you just need to understand it

#

I did it through some weird algorithm

#

that isn't the fastest in the land

#

but if you want my implementation, there it is

lost schooner Jul 2, 2024, 1:18 AM

#

I read the google docs on it, and it explained something about shadow space, but I don't know how to actually implement that

hardy lake Jul 2, 2024, 1:19 AM

#

what I did is slapped an extra 256 bytes off every allocation and filled it with some poison value

#

then when checking for corruption, I simply compare the pointer with said poison value

#

if they're the same, it checks the bytes around, and if either side have the poison value, then it yells at you

#

it isn't a fast or good implementation

#

but it's simple enough

#

and it works*
*for the one case that it actually covers

lost schooner Jul 2, 2024, 1:24 AM

#

I could always do what you did originally and just emulate the CPU's MMU, but that would be very very slow

hardy lake Jul 2, 2024, 1:24 AM

#

lost schooner I could always do what you did originally and just emulate the CPU's MMU, but th...

that's not what asan does

#

I had my understanding of its purpose borked

lost schooner Jul 2, 2024, 1:25 AM

#

yeah, it wouldn't catch what I'm looking for

lost schooner Jul 2, 2024, 7:14 AM

#

asan on GCC seems to have so many extra required symbols

#

and clangd doesn't like the compiler flag for ASAN

#

i get Unsupported option '-fsanitize=address' for target 'x86_64-frostyos' for every open C or C++ file

lost schooner Jul 2, 2024, 7:34 AM

#

I hopefully should have a basic KASAN implementation

#

It follows a similar concept to @hardy lake's implementation

#

very early boot triple fault

#

this is going to be annoying to debug

#

it appears to be the very first read that it triple faults on

#

I definitely stuffed something up

#

i was using -fsanitize=address instead of -fsanitize=kernel-address

#

it doesn't even detect anything

lost schooner Jul 2, 2024, 8:48 AM

#

ubsan is handling it before KASAN can, but it is a use after free

hardy lake Jul 2, 2024, 9:25 AM

#

lost schooner It follows a similar concept to <@841762704766140486>'s implementation

Don't forget to disable KASAN for the functions that implement KASAN

lost schooner Jul 2, 2024, 9:26 AM

#

I didnt forget

lost schooner Jul 2, 2024, 9:55 AM

#

I realised that my memset/memcpy are in assembly, so no KASAN. I decided it would be easier to quickly write fast versions in C (not C++) instead of trying to do some weirdness in assembly to get KASAN working.

#

nevermind, I'm not making a fast memcpy/memset in C. It can just be standard slow 1 byte operations

#

i tried to make it fast, but with ubsan and KASAN, GCC generated a 636 byte function

#

for memcpy that is

#

memset isn't much better

#

622 bytes

#

that is insane

lost schooner Jul 2, 2024, 10:21 AM

#

lost schooner I realised that my memset/memcpy are in assembly, so no KASAN. I decided it woul...

back to this, I decided to have the assembly functions available as memset_nokasan, and memcpy_nokasan for certain cases (such as copying from back to front buffer in the VGA stuff).

#

side note, I just noticed that electron apps use massive amounts of virtual memory, but barely any physical memory

#

this is amusing to see happen:

#

I have no clue how it happened

#

my memcpy must be dodgy

#

nope

#

just forgot to remove a divide by 8

lost schooner Jul 3, 2024, 12:48 AM

#

lost schooner ``` KERNEL PANIC! UBSan: type mismatch at /home/frosty515/dev/FrostyOS/FrostyOS/...

back to this, i discovered that the linked list node being used has been freed

#

KASAN isn't detecting it though

#

I must have missed something

#

finally:

ASAN Violation! Use After Free at ffffffff8004c322 whilst trying to read 8 bytes from ffffffff802c3350 on CPU 0

#

after doing some printf spam, i discovered that the object was freed in a prior call to PageManager::MapPages

#

is wasn't in that exact function, but it started from that function

lost schooner Jul 3, 2024, 2:10 AM

#

this is so weird. I can't work out why that object is being used after being freed

#

i never had any issues until I changed the PageManager

lost schooner Jul 3, 2024, 4:08 AM

#

i've just stepped through it with GDB, and I think I found the conditions for it to happen

#

PageManager::MapPages --> VirtualPageManager::UnfreePages with page count > 1

#

that second function is responsible for removing something from the free pages AVL tree

lost schooner Jul 3, 2024, 4:38 AM

#

what appears to be happening is that the wrong node in the free pages AVL tree is having a node in its linked list being deleted

#

I don't see how though

lost schooner Jul 3, 2024, 4:56 AM

#

i might have finally found the bug

#

the conditions are when an AVL tree node to be deleted has 2 children

#

The way that delete works is that the in-order successor's data is copied to the parent, and then the old node for the in-order successor is deleted

#

I was only copying the key, not the extraData as well

#

I don't know how I haven't noticed this before

#

alright, it gets further in boot now

#

KASAN has caught another use after free

#

looks like I forgot that kasan should be off for all of the heap related functions

#

I had temporarily disabled heap freeing from actually working so I could track down that bug, but I just had re-enable it, because otherwise uACPI uses all of the heap in a very short amount of time

#

now I am waiting for uACPI namespace init, which is taking ages with KASAN

#

finally uACPI init is done

#

I think my printf might be an issue

#

I might need to disable KASAN for parts of it

#

it looks like my new PageManager is working well

lost schooner Jul 3, 2024, 5:24 AM

#

lost schooner I think my printf might be an issue

its so bad that it takes around 10 seconds to create an initramfs item

#

I don't see individual characters being printed though

#

it all seems smooth

#

just big delays in between things being printed to the screen

lost schooner Jul 3, 2024, 5:26 AM

#

lost schooner it all seems smooth

that might be due to how the double buffering works

#

something is really slow when printing with double buffering enabled

#

i changed all the __attribute__(no_sanitize("address")) to __attribute__(no_sanitize("address","kernel-address"))

#

and added that attribute to a bunch of VGA functions

#

now it is much faster

#

and works well

#

time to re-enable ubsan and SMP

#

an ubsan type mismatch, which in this case is a nullptr dereference, followed by a triple fault

#

ohh

#

its whilst copying the AP trampoline

#

I just made the memcpy be the memcpy_nokasan version because that is in assembly

#

now it all works

#

ACPI shutdown and everything

#

I wonder what happens on real hardware...

#

its been a while since I last did a real hardware test

dusky orbit Jul 3, 2024, 5:48 AM

#

wow you are doing a lot of things recently

lost schooner Jul 3, 2024, 5:50 AM

#

yeah. It's school holidays, so I have more time

#

Anyway, it doesn't get as far on real hardware anymore

#

It freezes part way through loading one of the many SSDTs (maybe the 7th of 12?)

#

It might be the kernel heap getting too big

#

It is a fixed size at the moment

#

Because I couldn't work out how to properly make it dynamic as the VMM relies on it to allocated anything

#

Ill try increasing the size of it

dusky orbit Jul 3, 2024, 5:57 AM

#

ah that circular dependency

lost schooner Jul 3, 2024, 5:58 AM

#

I've had ideas on how to fix it, but nothing that is actually decent and will work reliably

dusky orbit Jul 3, 2024, 5:59 AM

#

I've gone with 2 heaps, one for vmm management structures (there's only 2 and theyre the same size, so slab it is), and then one for general use which uses the vmm.

#

the first one is just grabbing pages from the pmm

#

its not glorious, but it does the trick

lost schooner Jul 3, 2024, 6:00 AM

#

that sort of idea might work

#

my VMM has 2 different types, but they are different sizes. Doesn't matter though, my allocator is a special type of linked list. Just a series of blocks in one big chunk, linked together

#

I might try doing something similar to what you do

#

in the way that you have 2 heaps, one backed by the PMM, and the other by the VMM, not the actual type of allocator

#

Increasing the heap size to 8MiB instead of 4MiB seems to fix that issue

#

Now it gets stuck at the same place as before

dusky orbit Jul 3, 2024, 6:10 AM

#

lol that is also a solution

lost schooner Jul 3, 2024, 6:12 AM

#

only a temporary solution

#

it is time I actually fix my heap

lost schooner Jul 5, 2024, 5:54 AM

#

dusky orbit I've gone with 2 heaps, one for vmm management structures (there's only 2 and th...

I tried implementing something similar to this ^^^, and it all seemed to be going well until uACPI namespace load

dusky orbit Jul 5, 2024, 6:27 AM

#

uacpi is a good test suite for a kernel 😄

lost schooner Jul 5, 2024, 6:28 AM

#

yep

#

appears like I've got a dodgy block size

#

it says 32, when there is only actually 16 bytes

#

the header for each block happens to be 16 bytes, which suggests that I forgot to subtract that somewhere

#

or a merge with another block didn't go well

lost schooner Jul 7, 2024, 12:12 AM

#

At first I thought that the issue was something to do with the expansion stuff, but after further investigation, I suspect that the issue has been there all along, and because the heap was previously a fixed size so it never got full enough for me to notice the issue.

#

I'm tempted to fill many of the heap related functions with printfs so I can catch the issue without having to manually step through it with GDB. It will still be slow, but not as slow as GDB hopefully...

#

I say all the time that I don't want to rewrite, but what I have been doing lately is essentially a rewrite

#

I've redone the whole TTY interface, PageManager, now the heap. I plan on doing the PMM or core VFS next.

#

I redid the scheduler and IRQ system a couple of months ago when I started using SMP

#

Most of my old ACPI code was replaced by uACPI

#

Instead of starting completely from scratch, I'm redoing various parts in stages to still keep things somewhat stable.

lost schooner Jul 7, 2024, 4:53 AM

#

lost schooner At first I thought that the issue was something to do with the expansion stuff, ...

I finally found it, and it is related to the expansion stuff

#

I was right about maybe forgetting to subtracting the size of a block header

#

I had forgotten that when I expand the heap, that there are actually 2 new block headers: one for the block being allocated, and one for the remainder of the freshly allocated chunk

#

I just caught an odd deadlock occur

#

for the heap to expand, the VMM is called, which in some cases will cause a TLB shootdown IPI

#

the IPI info is heap allocated for some reason

#

which causes that CPU to get stuck if an IPI needs to be triggered when the heap is locked

#

the simplest, but probably not the correct solution is to just allocate the IPI info on the VMM heap

#

I'll do that for now until I come up with a better idea

lost schooner Jul 7, 2024, 5:19 AM

#

and the heap is working properly now

#

time to see if I can implement automatic shrinking

lost schooner Jul 7, 2024, 5:49 AM

#

this is very wrong:

lost schooner Jul 7, 2024, 6:09 AM

#

I think maybe I added too many printfs

#

my terminal is dying

#

I have found an odd bug

#

a block is allocated with kmalloc of a size of 1424

#

when that block is freed with kfree, it has a size of 0

#

I'm yet to verify what the block header says the size is on kmalloc

#

another issue which seems to appear at a similar time is that an item in the free list has its isFree flag set to 0

#

it appears just before

#

that flag changes while the block is in the free list

#

it is not removed and readded at all

#

it becomes incorrect after a heap-expanding kmalloc

#

the size of the block is also incorrect

#

i'll disable shrinking and see what happens so I can work out what I need to investigate

#

my terminal is once again dying

#

currently at 90% CPU usage

#

why does uACPI have to be so heavy on the heap

#

its been running for about 5 minutes now, and my kernel hasn't came crashing down yet

#

my heap isn't getting too fragmented

#

for the most part it seems the allocations are under 64 bytes

#

luckily KASAN is off at the moment

#

ubsan is still on, but it doesn't normally have much of a performance impact

#

I left it running for 10 minutes, and it is still going

#

nothing has changed on the screen

#

my heap is starting to get very big and fragmented now

#

its been over 20 minutes since it first started

#

it's taking longer and longer for each operation, because there are more blocks to print info about

#

it's still much faster than trying to use GDB

#

a very large number of allocations just happened

#

the amount of free blocks just shrank a lot

#

must have been lots of big allocations

#

finally it finished

#

only took 45 minutes

#

well now I know that the shrinking is the issue

#

I don't feel like dealing with that at the moment though

dusky orbit Jul 7, 2024, 7:36 AM

#

a couple of things that helped debugging my heap:

if you store management data intrusively, keep a copy elsewhere so you can compare when freeing so it if its been clobbered (may not apply to your code, and is kind of wasteful). You can assert() or whatever on alloc/free that your data is intact.
a page-heap mode, where each allocation gets a separate block of virtual memory with guard pages, and the return pointer is adjusted so that the if an access is attempted beyond the number of bytes requested, its in the guard page. Or you can do return the pointer at the start of the block, to cave underruns - less likely though.

lost schooner Jul 9, 2024, 5:44 AM

#

Those both seem like good ideas, but I don't think those would be useful at the moment. There is an issue with my shrinking which causes things to be weird.

#

I might start with verifying that heap is still valid after each operation so I can catch exactly when things fall apart

#

I might add some asserts in some places as well just to catch issues earlier

lost schooner Jul 9, 2024, 6:35 AM

#

this issue is so weird

#

the more I look into it, the more confusing it gets

#

I think I might take a break from this for a few days and work on other projects

lost schooner Jul 12, 2024, 5:40 AM

#

It's been almost 3 days now, so I think I might come back to this either today (if I feel motivated) or tomorrow

lost schooner Jul 13, 2024, 10:52 AM

#

And I didn't work on this yesterday or today

#

Oh well

#

I should have time tomorrow

#

I want to finally get this heap working properly

lost schooner Jul 14, 2024, 4:31 AM

#

time to return to osdev

#

The specific case where shrinking goes wrong is when a BlockHeader is being added to the free list in between 2 existing blocks

#

it only happens after a few expands and shrinks

lost schooner Jul 14, 2024, 4:57 AM

#

it seems to be happening after the first kmalloc after a shrink then expand?

#

so that kmalloc is responsible

#

actually, that kmalloc is causing the expand

#

it seems to break after clearing the freshly allocated block to 0. It isn't even part of the heap yet

#

my memset could be bad

#

I'm currently testing to see if it is, but my C-based memset/memcpy are really slow

lost schooner Jul 16, 2024, 11:11 AM

#

I forgot to explain what happens when I use the C variants of memset/memcpy

#

The short explanation is many bad things happen

#

Everything falls apart

#

I have no clue why

#

Something very odd is going on

#

I haven't actually worked on this since 2 days ago, and probably won't for a few more, but I will see how I go

#

When I do feel motivated to next work on this, I might disable SMP, and strip down the kernel a bit to try find the source of the issue

#

My userland port could prove useful

#

I'll need to make some changes to get it working again because of the TTY rewrite

#

I think I was going to treat userspace I/O as a serial interface

proud nova Jul 16, 2024, 12:52 PM

#

lost schooner I forgot to explain what happens when I use the C variants of memset/memcpy

They're probably trying to use something your kernel isn't properly configured for. Or something not available at all in kernels.
I would not be surprised if they're trying to use vector registers but you haven't added support for those in your kernel so it just breaks.

lost schooner Jul 16, 2024, 8:31 PM

#

proud nova They're probably trying to use something your kernel isn't properly configured f...

By C variant, I meant that I made the functions in C instead of in assembly. The only external code in my kernel is uACPI.

proud nova Jul 16, 2024, 8:32 PM

#

Depending on how you wrote it, GCC could still implicitly do things like add vector ops.
Or you have a bug. I don't know.

lost schooner Jul 16, 2024, 9:49 PM

#

I make sure to pass -mgeneral-regs-only. Its probably something dumb as it always seems to be

lost schooner Jul 19, 2024, 5:39 AM

#

I should be able to actually work on this at some point this weekend. I have time today, but I really don't feel like it

lost schooner Jul 24, 2024, 6:08 AM

#

I have decided it is time for me to return to this

#

it has been 10 days since I properly did something on this

#

I've decided to start off with fixing my userland port since all the TTY refactoring broke it

#

and I just discovered that my user page manager is broken

lost schooner Jul 24, 2024, 7:04 AM

#

i got it to build, with some messing around

#

but it seg faults

#

i think it is a global constructors problem

#

which means making a proper linker script

#

the .ctors and .dtors sections are empty

#

I forgot that linux uses .init_array and .fini_array instead

#

oh well

#

now I'm back to the heap issue

#

testing will be a little different as no uACPI

#

its a stack overflow due to what appears to be a rescursive function with no exit

#

oh I see the problem

#

for some reason an AVL tree entry in the VMM points to itself

#

to start with, I made the stupid mistake of redirecting the VMM heap functions to the normal heap

#

fixed that, but the issue still happens

lost schooner Jul 25, 2024, 4:38 AM

#

turns out I had forgot to update everything in the UserPageManager to use the VMM heap functions

#

it all works when it pretends to shrink the heap, but doesn't actually free the pages. When the pages are freed, a use after free seems to happen immediately after the first shrink

#

I finally found a bug

#

when I shrink the heap, I don't calculate for the case where the header being inserted to the free list gets merged with the previous block. When a block gets merged with the previous block, the prev variable gets set to the header variable, and I wasn't checking for that case in the shrinking.

#

Now that I do that, I seem to be getting a different problem

#

for some reason, there is a chunk whose size isn't page aligned

#

it is most likely due to incorrect setting of the isEndOfChunk header member, but I don't know for certain

lost schooner Jul 26, 2024, 8:37 AM

#

I think I finally found another bug

#

it was due to BlockHeaders not be cleared correctly

#

hopefully my heap is fixed now

#

my test is going to take a little bit due to all the debug printing

#

my ridiculous test was successful

#

but

#

it breaks during initramfs stuff

#

for anyone interested in the test, it can be found here

lost schooner Jul 26, 2024, 8:51 AM

#

lost schooner it breaks during initramfs stuff

just another memory not being cleared correctly

#

my heap appears to finally be fixed

#

time for a full kernel mode test

#

it doesn't work

#

which is fun

#

I'm starting to suspect my VMM

#

I have confirmed the heap works on its own, so my VMM is to blame I think

#

it probably doesn't like all this rapid allocations and free's

#

anyway, I've had enough of this for today

lost schooner Jul 30, 2024, 2:45 AM

#

its been 4 days, so I think I'll come back to this after school today

lost schooner Jul 30, 2024, 5:48 AM

#

I think the first step is running some tests on the virtual address space allocator in a separate userspace program

#

how that goes will determine what happens next

lost schooner Jul 30, 2024, 7:45 AM

#

actually

#

I could probably test in my userspace port of my kernel...

#

my virtual address space allocator doesn't require much so it should be pretty simple

lost schooner Jul 30, 2024, 8:04 AM

#

and the test seems to have succeeded

#

which is concerning

#

that means something else is broken

#

no, it did not succeed

#

seems like my test might actually be bad

#

I had forgot that everything is in pages not bytes

#

it still fails though

#

its my fault again

#

I was checking if the address is nullptr when the allocation range starts at 0

#

these numbers don't add up:

#

ignore the random rotated L in the top left

#

I've just switched from hyprland to i3, and I'm still working out some stuff

#

anyway, 196 + 24 + 28 + 280 does not equal 1048576

#

wait

#

those lists have multiple entries

#

I mean those nodes have lists with multiple entries

#

maybe it is just really fragmented

#

nope

#

those nodes all contain a list with one entry

#

well, they might explain why things are breaking in the normal mode

lost schooner Jul 31, 2024, 6:15 AM

#

I discovered something

#

after only 4 calls to AllocatePages, the size starts to become incorrect

#

nothing massive, but it is concerning

lost schooner Jul 31, 2024, 6:35 AM

#

that is very concerning

#

my Internal_SplitFreeBlock functions is broken

lost schooner Jul 31, 2024, 7:15 AM

#

actually, it is very inconsistent

#

sometimes that breaks, other times it is my UnfreePages function

#

actually, I'm almost certain it is my UnfreePages function

#

it completely destroys the FreePagesSizeTree

lost schooner Jul 31, 2024, 9:16 AM

#

I'm starting to wonder if my AVL tree code is dodgy

lost schooner Aug 2, 2024, 10:08 PM

#

Once again, its been almost 3 days

#

I'll have some time this afternoon to try sort this out

lost schooner Aug 3, 2024, 3:22 AM

#

I think it might actually be my AVLTree:deleteNode function

lost schooner Aug 3, 2024, 3:41 AM

#

I'm not 100% certain, but it seems like if a delete a node, everything in the left subtree is also deleted

#

after messing around with different ways of displaying the contents of the tree, I'm almost certain that is exactly what is happening

#

if I disable self-balancing it seems to work

#

the tree turns into a mess, but it works

#

the self-balancing code happens to be the part of the implementation that I barely understood when I made, and have no clue how it works now

#

so this will be fun

#

I noticed that it was different between the insert and delete functions, and the insert works so I tried copying it to the delete

#

and I got a nullptr dereference caught by ubsan instead of an irritating segfault

#

that is why ubsan is amazing

lost schooner Aug 3, 2024, 4:28 AM

#

I managed to fix it

#

another part of the kernel now fully tested and working

#

i tried going back to the full version of my kernel, and uACPI killed the heap

#

I'm back to the same old bug of an entry in the free list having all its data cleared

#

I think I'll try get KASAN working again

#

first of all, I'll update uACPI because its been quite a while

lost schooner Aug 7, 2024, 8:05 AM

#

I have been busy with various things lately, but I should hopefully have time to return to this tomorrow afternoon

#

I was hoping I could today, but I don't have enough time

lost schooner Aug 8, 2024, 4:26 AM

#

alright

#

it is time for me to return

lost schooner Aug 8, 2024, 4:38 AM

#

lost schooner I'm back to the same old bug of an entry in the free list having all its data cl...

back to this

#

it seems like the clearing of freshly allocated memory for the heap is causing the issue

#

which points to the VMM being dodgy still

#

I wonder if it is the PageManager I rewrote somewhat recently

#

i think I might try testing it in the userland port again

#

its not going to be as simple to get up and running though

#

this is due to all the page manager functions being overridden by wrappers around the mmap family of linux syscalls in usermode

lost schooner Aug 8, 2024, 5:17 AM

#

finally managed to get something I can test

#

I have made an empty test PageTable and PhysicalPageFrameAllocator just so I can test this

#

for now, I'm just letting it run through the whole stress test without any verifies to make sure there isn't any obvious problems

#

I can hear my CPU fan getting faster and faster the longer this runs

#

alright this is a waste of time

#

I'll start implementing some actual tests

#

i disabled all the extra debug printing, and the early test is successful

lost schooner Aug 8, 2024, 5:30 AM

#

lost schooner I'll start implementing some actual tests

now I'll actually do this ^^^

#

wait

#

I don't actually see how the PageManager could be causing this

#

it could actually be the PMM

#

something at one virtual address being written also changing something at another virtual address sounds like it could be the PMM

#

the PageManager only deals with virtual addresses

#

so either my page mapping code is wrong or my PMM is wrong

#

im going back to kernel mode to investigate this possibility

lost schooner Aug 8, 2024, 6:01 AM

#

when I try to manually step through the page tables, both pages were mapped to 0, but QEMU's info tlb command says they are mapped to different physical addresses

dusky orbit Aug 8, 2024, 6:03 AM

#

dirty tlb entries?

lost schooner Aug 8, 2024, 6:06 AM

#

I'm probably manually reading them wrong. As for QEMU's command, I remember seeing somewhere that it doesn't actually read TLB entries, but instead just reads the raw tables in memory.

#

also, just to eliminate the possibility of a bad memset, I tried both my optimised asm version and my simple C version.

#

exact same results

dusky orbit Aug 8, 2024, 6:09 AM

#

one less variable to worry about at least

lost schooner Aug 8, 2024, 6:10 AM

#

yeah

#

my heap could still be broken, but in a less obvious way

#

I re-enabled KASAN, and it seems to have already caught something

#

well, ubsan caught the problem first, but still

#

it seems like a shadow space access

#

by the AVLTree implementation

#

its a VMM heap object

#

it seems to be from a findNodeOrHigher call from VirtualPageManager::UnfreePages

#

oops

#

its my fault

#

I forgot a very important few lines in kmalloc_vmm and kcalloc_vmm to increase the size for KASAN allocations

#

KASAN doesn't catch anything

lost schooner Aug 9, 2024, 6:12 AM

#

I starting looking into this further, and both pages are mapped to physical page 0, which is concerning

#

maybe the PMM is the issue...

#

I already found 1 bug in the PMM, which is that the locks don't actually works due to some weirdness

#

for simplicity, the locks are disabled between calls to EarlyInit and FullInit

#

and I had forgetten about the case where FullInit returns early

#

I added a check to the PMM to detect when the first page of memory (0x0000) is attempted to be allocated

#

and the first call to HeapExpand after a HeapShrink triggers it

#

I found the problem

#

its so stupid

#

I was unmapping the page before getting its physical address

#

it works through uACPI stuff, but fails towards the end of the initramfs

#

but, the PMM check is not triggered this time

#

same issue of a free list entry being all 0, but it is in the VMM heap

#

in a TLB shootdown

#

the call to HeapAllocator::Verify() is in a different place now

#

actually, it is the same thing, except with the VMMHeapExpand instead of HeapExpand

#

I found another bug

#

I was doing to_HHDM instead of hhdm_to_phys in VMMHeapShrink

#

the same physical memory page is being allocated twice

#

finally

#

I found it

#

for some unknown reason

#

in my PMM FindFreePages (not to be confused with FindFreePage) function, specifically when the requested count is 1, the returned index would get multiplied by 8

#

I don't know when I did that, but it would have been over a year ago

#

it all finally works

#

now I have to work out what I was doing before all this nonsense

#

I do know I was going to add serial support

#

I also know that my ACPI stuff needs improving

marsh gazelle Aug 9, 2024, 7:02 AM

#

no uACPI?

#

or do you have your own few table parsing functions

lost schooner Aug 9, 2024, 7:03 AM

#

marsh gazelle no uACPI?

have a read up like 20 messages

marsh gazelle Aug 9, 2024, 7:04 AM

#

i see KEKW
the usual outcome

#

uacpi is more of an early allocator testing api atp

lost schooner Aug 9, 2024, 7:04 AM

#

I've used uACPI for a while

#

i have ACPI shutdown working

#

speaking of which, I'm going to clear out some of the debug printing and run a real hardware test

marsh gazelle Aug 9, 2024, 7:05 AM

#

good luck with that

lost schooner Aug 9, 2024, 7:18 AM

#

stupid heap is causing problems again

#

supposedly an item in the free list has a size of 0

lost schooner Aug 9, 2024, 7:35 AM

#

got that fixed, now the problem is that a end of chunk block has the incorrect address or size (most likely size)

dusky orbit Aug 9, 2024, 8:23 AM

#

damn you're running into a few memory allocator bugs here 😦

lost schooner Aug 13, 2024, 5:40 AM

#

and I accidently took another 4 day break

#

after seeing more memory allocation bugs after thinking I had it all working, I kinda just lost all motivation for a few days

marsh gazelle Aug 13, 2024, 5:46 AM

#

https://github.com/FrostyOS-dev/FrostyOS/blob/master/kernel/src/Memory/kmalloc.cpp#L114
if i‘m not misreading this you’re allowing blocks of size 0 being created here?

lost schooner Aug 13, 2024, 5:46 AM

#

marsh gazelle <https://github.com/FrostyOS-dev/FrostyOS/blob/master/kernel/src/Memory/kmalloc....

old code

marsh gazelle Aug 13, 2024, 5:47 AM

#

oops

#

where can I find the new one

lost schooner Aug 13, 2024, 5:47 AM

#

most recent commited code is in the testing branch, but most recent changes are still local

#

all of the rewrites, and the userspace port is still local

#

since I can't be bothered to commit it

marsh gazelle Aug 13, 2024, 5:50 AM

#

is the part of the allocator that’s making issues up to date?

lost schooner Aug 13, 2024, 5:50 AM

#

no

#

well, I found a bug

#

something just tried to allocate 32768 bytes on the heap, which did not go well

lost schooner Aug 13, 2024, 5:57 AM

#

lost schooner well, I found a bug

it was a stupid bug related to the 0 sized block

lost schooner Aug 13, 2024, 5:59 AM

#

lost schooner something just tried to allocate 32768 bytes on the heap, which did not go well

back to this

#

it seems like it was a poorly timed interrupt during PS/2 controller init

#

an interrupt gets fired whilst reading the PS/2 config byte

#

the good thing is that uACPI init is successful with the new heap now

#

which means my heap and VMM actually seem to work

marsh gazelle Aug 13, 2024, 6:08 AM

#

🎉

lost schooner Aug 13, 2024, 6:12 AM

#

i am currently testing to see what happens when I disable the PS/2 stuff

#

it freezes when creating the first file in TempFS for something from the initramfs

#

I disabled the initramfs just to see if it gets to ACPI shutdown

#

and

#

it works

#

which I am very happy about

#

I'll deal with the tempfs nonsense another time

#

at least I know that uACPI in FrostyOS seems to be working well

#

I am going to do some cleanup and then get this code into the testing branch

lost schooner Aug 13, 2024, 6:59 AM

#

and the new code is all in the testing branch

#

that can be found here

hollow trail Aug 13, 2024, 7:05 AM

#

nice

lost schooner Aug 13, 2024, 7:16 AM

#

almost at 30k loc:

hollow trail Aug 13, 2024, 7:23 AM

#

damn

lost schooner Aug 13, 2024, 8:02 AM

#

I think I just have to some further clean-up and implementation of stuff, then I might be able to finally merge all this ACPI stuff into master

#

then I'll be able to move on to other things

lost schooner Aug 13, 2024, 9:25 AM

#

it looks the like the legacy PIT doesn't exist in the ACPI namespace on QEMU. running uacpi_find_devices with hid equal to PNP0100 resulted in no devices being detected

lost schooner Aug 13, 2024, 10:04 AM

#

doesn't really matter

hollow trail Aug 13, 2024, 10:24 AM

#

linux assumes pit exists unconditionally on x86

lost schooner Aug 13, 2024, 10:29 AM

#

I know that it exists, its just not in ACPI namespace

hollow trail Aug 13, 2024, 11:24 AM

#

yeah qemu doesnt have it

#

we talked about it here before

lost schooner Aug 13, 2024, 11:44 PM

#

my scheduler is being painful

#

there seems to be some kind of race condition somewhere causing problems

#

I've seen 2 different errors during the process of adding a processor to the scheduler

#

and now a seemingly random GPF with an error code of 0x45a0

#

I did manage to get ACPI reboot working via uACPI

#

here is one of the issues:

#

from the Scheduler::AddProcessor function

#

all the CPUs have LAPICs in their data structures

#

so this doesn't make sense

#

the BSPs TSS looks like it isn't being cleared before being used, but other than that, nothing looks odd

lost schooner Aug 14, 2024, 12:03 AM

#

lost schooner here is one of the issues:

this error seems to happen on different CPUs at different times

#

there is no consistency

#

actually, there is some

#

it seems like it is always the last one added, but that could be wrong

#

nevermind

#

they all init at basically the same time

celest sierra Aug 14, 2024, 12:38 AM

#

lost schooner so this doesn't make sense

maybe your lapic detection code doesnt make sense?

#

though idk why you'd try and detect that

lost schooner Aug 14, 2024, 12:52 AM

#

celest sierra though idk why you'd try and detect that

its just a safety check to see if the processor data structure contains a pointer to its LAPIC so there isn't just a random page fault

lost schooner Aug 14, 2024, 4:17 AM

#

haven't seen this one before:

lost schooner Aug 14, 2024, 4:32 AM

#

i found a bug

#

the LAPIC timer for that CPU was running prematurely, because I never disabled it

#

that wasn't it

#

it is still running when it shouldn't be

#

With this code below, its almost like the function call happens before the if statement occassionaly

if (info->current_thread->GetCPURegisters() == nullptr) {
    PANIC("Thread does not have CPU registers.");
}
#ifdef __x86_64__
LAPIC->InitTimer();
#endif

dusky orbit Aug 14, 2024, 6:11 AM

#

between the gpf and the pointer weirdness I wonder if its memory corruption

lost schooner Aug 14, 2024, 6:20 AM

#

i'm starting to think that

#

pointers being nullptr for specific instructions, so when I go to check they are fine

lost schooner Aug 15, 2024, 4:44 AM

#

I have no idea how I'm going to debug this

#

the GPF is always in the assembly common interrupt handler

#

when it occurs is seemingly random, with a seemingly random error code

lost schooner Aug 15, 2024, 4:53 AM

#

lost schooner and now a seemingly random GPF with an error code of 0x45a0

a error code of 0x45a0 becomes GDT selector index 0x8B4

#

that is definitely bogus

lost schooner Aug 15, 2024, 4:57 AM

#

lost schooner haven't seen this one before:

this is looks like the idle task failed to resume from an interrupt

#

another isr handler bug...

#

first of all, I'm going to try work out why the LAPIC timer is being initialised early sometimes

lost schooner Aug 15, 2024, 5:34 AM

#

one problem I suspect is that interrupts (for whatever unknown reason) are happening during the process of adding the processor to the scheduler and getting something happening on it

#

here is the latest page fault:

#

another pointer magically becoming nullptr

dusky orbit Aug 15, 2024, 10:26 AM

#

lost schooner one problem I suspect is that interrupts (for whatever unknown reason) are happe...

thats an easy hypothesis to test

#

does the issue also occur when running without smp?

lost schooner Aug 15, 2024, 10:44 AM

#

Well obviously no, because that function only gets called when there is more than 1 CPU

lost schooner Aug 15, 2024, 11:14 AM

#

I do wonder if I let it run enough times without SMP, I might catch something

lost schooner Aug 17, 2024, 9:07 AM

#

When I am motivated to work on this, I might just add a bunch of checks in many places to make sure everything is behaving

lost schooner Aug 17, 2024, 9:48 AM

#

I've been thinking about this a bit more, and a rewrite is definitely on my mind

#

So much code is based off the horrible original design

#

It all sucks

#

The heap works, but is messy

#

The VFS is barely functional, very slow, and just a complete disaster

#

The PMM is basically the original code

#

The VMM is a mess and not actually that fast

#

The scheduler, especially after implementing SMP, is a complete disaster

#

It's all over the place

#

The only somewhat ok code is the basic graphics stuff and the TTY stuff, which I just recently rewrote

#

I'll think about this some more overnight, but I'm almost certain I want to basically start from scratch.

#

I don't know when I'll actually do it if I choose to, but it will happen soon

#

I've got a very busy week IRL coming up, so I'll see how I go

sand scaffold Aug 17, 2024, 10:24 AM

#

good luck

lost schooner Aug 17, 2024, 11:26 AM

#

Thanks

hardy lake Aug 17, 2024, 3:13 PM

#

lost schooner I'll think about this some more overnight, but I'm almost certain I want to basi...

do it

#

and if you do, then gl

lost schooner Aug 17, 2024, 10:27 PM

#

After thinking about it some more, I'm going to do it

#

I'll make a start today, but I won't go too far without planning it out properly

hardy lake Aug 17, 2024, 10:35 PM

#

when you make your vmm

#

it should use the working-set model

hollow trail Aug 17, 2024, 10:40 PM

#

lost schooner I'll make a start today, but I won't go too far without planning it out properly

Is this your first rewrite?

hardy lake Aug 17, 2024, 10:41 PM

#

it's his first

#

which he's had since he started osdev afaik

#

looking forward to see how this one goes

hollow trail Aug 17, 2024, 10:46 PM

#

I guess not the last then

lost schooner Aug 17, 2024, 11:12 PM

#

hardy lake it should use the working-set model

I was planning on doing that

lost schooner Aug 17, 2024, 11:12 PM

#

hollow trail Is this your first rewrite?

Yes

hardy lake Aug 17, 2024, 11:12 PM

#

here's a paper

#

on it

lost schooner Aug 17, 2024, 11:13 PM

#

Thanks

#

I'm going to make sure I make everything thread safe to start with

#

Hopefully that will make it so I have less issues

hardy lake Aug 17, 2024, 11:15 PM

#

hopefully

#

gl

#

here's a really good implementation of a page replacement algorithm

cerulean obsidian Aug 17, 2024, 11:15 PM

#

hardy lake here's [a paper](https://dl.acm.org/doi/pdf/10.1145/363095.363141)

ip grabber

hardy lake Aug 17, 2024, 11:15 PM

#

well if it is an ip grabber, it's fadanoids

#

#virtual-memory message

#

I got the link from there

hardy lake Aug 17, 2024, 11:19 PM

#

cerulean obsidian ip grabber

this is an example of an ip grabber REDACTED

#

(don't click that)

#

(unless you want me to have your IP)

cerulean obsidian Aug 17, 2024, 11:19 PM

#

maybe dont send an ip grabber here

#

probably for the best

#

i doubt anything bad will happen but its not the kind of thing that belongs here id say

lost schooner Aug 17, 2024, 11:24 PM

#

hardy lake (unless you want me to have your IP)

you wouldn't find much from my IP

hardy lake Aug 17, 2024, 11:24 PM

#

I assume you use a VPN or proxy

lost schooner Aug 17, 2024, 11:24 PM

#

no, my IP location is just extremely inaccurate and I don't have any open ports on my router

#

anyway

#

I've have some free time today, so I'll make a start

#

the old code is here now

#

and the new code is at where the old code was

lost schooner Aug 18, 2024, 12:09 AM

#

I got it to compile after a bunch of messing around

#

a decided to copy over all of the basic kernel libc

#

I had to basically disable most of it though

#

and it seems to run

#

I just have 0xE9 debug printing at the moment just to make sure things work

lost schooner Aug 18, 2024, 12:44 AM

#

I now have all the limine requests that I need working

lost schooner Aug 18, 2024, 1:43 AM

#

well this is interesting:;

hardy lake Aug 18, 2024, 1:47 AM

#

lost schooner well this is interesting:;

probably the glyph mask being broken

lost schooner Aug 18, 2024, 1:51 AM

#

there we go:

#

I forgot the bytes are backwards in the font

#

for whatever reason, the bytes are bottom to top

#

I can't be bothered to fix that

dusky orbit Aug 18, 2024, 6:47 AM

#

very nice, good luck on the rewrite

lost schooner Aug 18, 2024, 7:04 AM

#

I had to have a break for a few hours, but I seem to have some proper framebuffer printing working:

marsh gazelle Aug 18, 2024, 7:05 AM

#

looks promising, good luck frosty

lost schooner Aug 18, 2024, 7:06 AM

#

Current todo list (in order):

proper TTY stuff to link framebuffer printing and debug printing

#

GDT

#

IDT/ISRs

#

after that, I'll need to go down either the scheduler route or the memory management route

#

I'm not sure which to do first

#

I need to make sure the scheduler is almost completely independent of the VMM

marsh gazelle Aug 18, 2024, 7:09 AM

#

then why not make them in parallel

lost schooner Aug 18, 2024, 7:26 AM

#

no, that is just asking for problems

#

I think I'll do scheduler first, but I'll need to think of a way to store thread metadata without using the heap at all

#

actually, my old implementation did that

#

I have no idea what I'm going on about

red swift Aug 18, 2024, 7:31 AM

#

lost schooner I think I'll do scheduler first, but I'll need to think of a way to store thread...

i did my entire fancy smp scheduler before i did my heap

lost schooner Aug 18, 2024, 7:32 AM

#

I liked how the rewrite of the TTY interface in my old kernel used backends, so I think I'll do that again

red swift Aug 18, 2024, 7:32 AM

#

you just give the scheduler a hunk of memory initialized as a thread struct and it deals in those

#

it doesnt care where the memory came from

#

doesnt allocate or free it itself

lost schooner Aug 18, 2024, 7:32 AM

#

I intend to do something like that

#

a scheduler that relies on the heap is a bad scheduler

red swift Aug 18, 2024, 7:33 AM

#

yes

#

dont do this for instance https://github.com/klange/toaruos/blob/master/kernel/sys/process.c#L660

#

oh nvm

#

that doesnt allocate a node

#

theres two functions with same name one does and one doesnt

lost schooner Aug 18, 2024, 7:47 AM

#

I'll be using intrusive linked lists so that the thread class contains pointers to the previous and next threads in the lists

#

no allocation necessary

red swift Aug 18, 2024, 7:48 AM

#

do this https://github.com/xrarch/mintia2/blob/main/OS/Executive/Ke/KeDispatch.jkl#L325

lost schooner Aug 18, 2024, 7:48 AM

#

thats some unusual syntax

red swift Aug 18, 2024, 7:48 AM

#

https://github.com/xrarch/newsdk/blob/main/Rtl/Headers/Rtl.hjk#L44

#

circular list macros, no need to even branch

lost schooner Aug 18, 2024, 7:50 AM

#

oh I see

#

thats a good idea

lost schooner Aug 18, 2024, 8:43 AM

#

lost schooner Current todo list (in order): 1. proper TTY stuff to link framebuffer printing a...

now done

#

I had to implement basic global constructor calling because of vtables, but it works

lost schooner Aug 18, 2024, 9:12 AM

#

stupid gnupg was being a pain so it took a while to successfully commit it, but finally I can move on to the GDT

lost schooner Aug 18, 2024, 9:45 AM

#

and a simple GDT is done

#

took a bit longer because I forgot to set the segment type in the access byte, and it took me a while to realise what I had stuffed up

#

I'll add proper system segment support when a make the TSS a little bit later

#

next on the list is IDT/ISRs, which will be a task for tomorrow

hollow trail Aug 18, 2024, 9:56 AM

#

red swift <https://github.com/xrarch/newsdk/blob/main/Rtl/Headers/Rtl.hjk#L44>

What is NOTHING

red swift Aug 18, 2024, 9:56 AM

#

hollow trail What is NOTHING

the state of NON-BEING.

hollow trail Aug 18, 2024, 9:59 AM

#

Why is it needed there

sand scaffold Aug 18, 2024, 9:59 AM

#

hollow trail What is NOTHING

no op token

#

it splits up expressions

#

if you do like

RtlAddUlongToUquad(myquad, 5)
RtlAddUlongToUquad(myquad, myotherquad)

#

its going to expand to

NOTHING (myquad)^.Quad += (5)
NOTHING (myquad)^.Quad += (myotherquad)^.Quad

hollow trail Aug 18, 2024, 10:01 AM

#

Soo what happens exactly if that nothing is gone

sand scaffold Aug 18, 2024, 10:01 AM

#

wait bad example

red swift Aug 18, 2024, 10:01 AM

#

hollow trail Soo what happens exactly if that nothing is gone

parses as a function call of the expression on the end of the previous line lol

#

its a problem caused by lack of semicolons

#

Lack of Semicolons Considered Harmful

#

a = b
(c) = 5

ends up parsing as like

a = b(c)

then it sees the = token and goes huh?

#

its great

hollow trail Aug 18, 2024, 10:02 AM

#

I see, guess semicolons are not that bad afterall

marsh gazelle Aug 18, 2024, 10:58 AM

#

red swift do this <https://github.com/xrarch/mintia2/blob/main/OS/Executive/Ke/KeDispatch....

may I ask what the ^ does?

red swift Aug 18, 2024, 10:58 AM

#

ptr deref

hollow trail Aug 18, 2024, 10:59 AM

#

marsh gazelle may I ask what the ^ does?

red swift Aug 18, 2024, 10:59 AM

#

its left-associative so ^. is the same as ->

marsh gazelle Aug 18, 2024, 11:01 AM

#

ah got it

lost schooner Aug 19, 2024, 7:25 AM

#

my IDT worked first try which is nice

#

very unlike last time...

#

time to make a proper kernel panic with stack tracing

lost schooner Aug 19, 2024, 9:02 AM

#

that is done

#

I discovered a couple of bugs with my framebuffer printing in the process, but it all works now

#

I might make a PMM next

#

I think I'm just going to go for a free list based version this time instead of a bitmap

#

this also means it is time to do something with the memory map limine gives me

#

QEMU really does like to place stuff randomly:

#

this is the current plan for after the PMM (in order):

page mapping
IRQs (legacy PIC and APIC) + SMP startup
scheduler
VMM (using the working-set model, so a lot more complex this time)
kernel heap

lost schooner Aug 19, 2024, 10:41 AM

#

the PMM seems to initialise without any issues

#

since I'm lazy, I'll use GDB to inspect everything

#

after 1 stupid bug, the init works

#

time to test allocate/free

lost schooner Aug 19, 2024, 11:04 AM

#

and the test works for allocation sizes of one page

#

tomorrow I'll test sizes larger than one page

lost schooner Aug 20, 2024, 10:22 AM

#

I did a small amount of testing in the little amount of time I had today, and found some bugs, still tracking down others.

lost schooner Aug 22, 2024, 9:27 PM

#

I probably won't be able to work on this at all until sunday

lost schooner Aug 25, 2024, 1:57 AM

#

finally I can make some progress

#

it looks like the current bug is a double allocation of the same address, resulting in a double free

#

Fixed a simple bug, and it now works fully for 10 million iterations of the test I made

#

currently testing 100 million, and it takes about 1-2 seconds for every 1 million iterations

#

this is not worth the time. It works fine. no freezes or random exceptions

#

and it also works for allocating/freeing regions larger than 1 page

#

well the PMM is done now

#

here is the test for anyone that is curious

lost schooner Aug 25, 2024, 2:28 AM

#

lost schooner this is the current plan for after the PMM (in order): 1. page mapping 2. IRQs (...

ok, back to this

#

so page mapping is next

lost schooner Aug 25, 2024, 4:47 AM

#

for some stupid reason I thought it would be a good idea to support the case where 1GiB pages exist, but 2MiB pages don't, even though that shouldn't be possible

lost schooner Aug 25, 2024, 5:52 AM

#

I'm also adding PAT support

lost schooner Aug 25, 2024, 6:43 AM

#

finally ready for the first test

#

and it is triple faulting

#

page fault leading to a double fault

#

well, I don't think CR3 should be 0...

lost schooner Aug 25, 2024, 7:56 AM

#

finally

#

it works

#

the framebuffer should be mapped as WC

#

everything else as WB

#

the mapping seems a little slow, so I'll investigate that a bit

#

I found a bug relating to the framebuffer mapping, but other than that, I have no idea

#

could just be QEMU not liking 1GiB pages

dusky orbit Aug 25, 2024, 8:11 AM

#

1G pages should make things faster, not slower

lost schooner Aug 26, 2024, 6:52 AM

#

well I found a problem

#

it seems to be using standard pages instead of 2MiB pages for some reason

lost schooner Aug 26, 2024, 8:44 AM

#

wow, that was a lot of bugs

#

I'm trying to make an algorithm to map pages using the most optimal page sizes, but it is not as easy as I hoped

#

I have no idea why, but it feels like after the last wake from suspend, electron apps are being very slow and unresponsive

#

my whole window manager just froze for a moment

#

odd

#

anyway...

lost schooner Aug 26, 2024, 9:15 AM

#

finally found another annoying bug

#

I was forgetting to not set the PAT bit for higher page table levels, which is PageSize

#

stupid mistake

#

the slowness should also be gone now

#

There is one more bug relating to the range of physical addresses that are HHDM mapped

#

the bug is from something in this line:

x86_64_MapRegionWithLargestPages(g_Level4Table, (uint64_t)fb_phys + fb_size, GiB(4) - ((uint64_t)fb_phys + fb_size) - KiB(4), 0x0800'0003); // Present, Read/Write, No Execute

#

it should be mapping from the end of framebuffer to 4GiB, but it maps one page short

#

oh wait

#

I'm so stupid

#

why am I subtracting 4KiB

#

just found another bug where a <= was a < for some reason

#

now that I have removed all the debug printing, start-up is fast

#

I think this means simple paging is done

#

I'll do a little clean-up and then commit this

#

and I get a GPF

#

how

#

its caused by enabling kvm

#

its on a wrmsr instruction

#

it is for writing the PAT

#

stupid mistake again

#

setting the index instead of raw value for each of the PAT types

#

at least it works now

#

since I'm bored, I'll do a quick real hardware test

#

first one of the rewrite

#

instant page fault

#

dodgy memory map entry?

#

its a nullptr dereference when attempting to read a memory map entry

#

this makes no sense

#

the PMM initialises successfully, but when the memory map is attempted to be used later, it is magically nullptr

lost schooner Aug 26, 2024, 10:05 AM

#

This is an interesting memory map:

#

it took so long to get that photo because the stupid camera wouldn't focus and my laptop screen is ridiculously reflective

#

the white spot at the bottom is just from my phone screen being bright

#

anyway, I've had enough of this for today

#

I was hoping to make a start on the IRQ interface this afternoon, but that didn't happen

#

oh well

hardy lake Aug 26, 2024, 10:30 AM

#

lost schooner the bug is from something in this line: ```cpp x86_64_MapRegionWithLargestPages(...

your flags look funny

lost schooner Aug 26, 2024, 10:53 AM

#

Low 12 bits (13 for 2MiB and 1GiB) are the low flags, lower 12 bits in the high 16 bits are the high flags

#

Little bit weird, but it works

lost schooner Aug 27, 2024, 5:48 AM

#

I might try to quickly get ubsan working

lost schooner Aug 27, 2024, 6:21 AM

#

it causes an a page fault, followed by a second page fault

#

I've been trying to work out how to get proper stack tracing inside an interrupt, and I have determined that GDB hates me

#

the stack trace seems to work until GDB tries to do it

#

I tried looking in other kernels to see what they do but that didn't help

#

I couldn't find it in linux

#

and the too simpler ones I found, managarm and astral, just seem to overwrite rbp in some way

#

looks like obos does the same as what I try do

#

@hardy lake do stack traces from inside interrupts work properly for you in GDB?

#

wait

#

I hadn't even enabled ubsan

#

huh

#

it works no issues

lost schooner Aug 27, 2024, 9:24 PM

#

lost schooner it works no issues

As in the page faults don't occur

#

The stack trace is obviously still messed up

lost schooner Aug 30, 2024, 7:11 AM

#

I've decided to stop procrastinating and just sort this problem out

#

it works in qemu perfectly fine with KVM either off or on with as much RAM as qemu would allow, so I don't see why real hardware wasn't liking it

#

and it works

#

on real hardware now

#

I don't know what I changed, but it works

#

why have I been procrastinating

#

it works on real hardware no issue

#

i guess irq system time

#

i wonder if ubsan is making it work...

#

yep

#

it doesn't work when ubsan is off

#

that is strange

#

it isn't irq system time then

#

why does this happen

#

its the same thing where one of the pointers to a memory map entry in the array is becoming nullptr

#

wait no

#

this is weird

#

on real hardware I get a nullptr dereference trying to read from the array of pointers to memory map entries, but on QEMU with KVM I get a page fault inside a page fault

#

ohhh

#

I thinking I'm starting to understand

#

the mapping ranges being fragmented like this was the first clue:

#

and then I discovered that the second page fault on QEMU was whilst trying to read the __stack_chk_guard, which happens to be in the gap between the last 2 rw regions

#

I think I fixed it

#

I wasn't page aligning the addresses of the start or end of the different kernel sections

#

and adding the ubsan code probably just shifted stuff enough to make it an issue

#

but by enabling ubsan, that changed a bit, so it worked again

#

doesn't work on real hardware unfortunately

#

the real hardware bug is from something else

#

Interesting:

#

the PMM initialises successfully using it though

#

interrupts are disabled, so there is no possiblity that my ISR code is causing this (not that it should be a problem, but just eliminating it)

#

after adding an extra assert in a different place as well, neither catch anything, but there is a page fault on the first read from the memory map after the second assert

#

interesting, the only flag set (other than bit 1, which is always 1) is the resume flag

#

I have no idea what that flag does

#

its related to debug exceptions. why is it set suddenly?

dusky orbit Aug 30, 2024, 12:13 PM

#

lost schooner it doesn't work when ubsan is off

possible race condition? ub-chan adds hooks everywhere and so it slows things down considerably.

lost schooner Aug 30, 2024, 9:12 PM

#

dusky orbit possible race condition? ub-chan adds hooks everywhere and so it slows things do...

How could there be a race condition if only 1 CPU is running?

lost schooner Aug 30, 2024, 10:32 PM

#

The only thing I can think of for that stage in boot is the PMM being dodgy

#

Potentially similar deal to the mapping bug

#

I'll look into that this afternoon, when I actually have time to work on this

lost schooner Aug 31, 2024, 3:51 AM

#

I couldn't see anything which might cause that

sand scaffold Aug 31, 2024, 4:47 AM

#

lost schooner How could there be a race condition if only 1 CPU is running?

race condition among threads on that CPU? unsure what you're asking

lost schooner Aug 31, 2024, 4:51 AM

#

That could be possible if my kernel even had the concept of threads...

#

I understand that interrupts could mess with things, but I don't have any IRQs configured, and if one were to fire, it would just immediately cause a kernel panic, because I don't have any form of IRQ handling.

hardy lake Sep 2, 2024, 10:14 AM

#

idk maybe try checking to see if all memory is zeroed

lost schooner Sep 2, 2024, 10:20 AM

#

I just have no motivation for this project at the moment, so I'll look into it more eventually

hollow trail Sep 2, 2024, 10:21 AM

#

daaamn

#

F

lost schooner Sep 2, 2024, 10:22 AM

#

It also doesn't help that I've got a bunch of school related stuff coming up and I'm sick once again

hollow trail Sep 2, 2024, 10:22 AM

#

get well soon!

lost schooner Sep 6, 2024, 9:58 AM

#

I'm not that sick anymore, so I'm starting to get some motivation for this, but I don't really have much time until after Wednesday

#

due to stupid school

#

but, the school term is almost over, so I'll have a lot more time soon

#

anyway, I'm going to try my best to ignore this project until then so I can focus on school ~~and minecraft~~

lost schooner Sep 11, 2024, 6:37 AM

#

finally

#

I can return to this

#

i have more motivation now that I'm not thinking about school

#

time to try work out why this real hardware bug is happening

dusky orbit Sep 11, 2024, 6:44 AM

#

welcome back

#

is it the same bug as before?

lost schooner Sep 11, 2024, 6:45 AM

#

I assume so

#

About to run a test

#

240 package updates

#

fun

#

qemu being responsible for 85 of those

#

it just hangs

#

not even an error message

#

wait

#

the cursor is still on screen

#

i think the usb decided to be special and just disconnect itself half way through loading the kernel file

#

it is a horrible usb

#

ok, so same problem as before

#

I added some debug printing to the PMM and there is no double allocations

dusky orbit Sep 11, 2024, 7:05 AM

#

lost schooner its related to debug exceptions. why is it set suddenly?

I dont know if its still relevant, but I'm thinking about this - do you have any usage of iret anywhere?

#

I'm wondering if perhaps its getting garbage data and thats why that flag ends up set, thats a wild guess though

lost schooner Sep 11, 2024, 7:06 AM

#

That has also been on my mind. The only use is in my interrupt handler

#

when interrupts shouldn't even be happening

dusky orbit Sep 11, 2024, 7:07 AM

#

what about an exception though?

lost schooner Sep 11, 2024, 7:07 AM

#

and if they do, its an instant panic

dusky orbit Sep 11, 2024, 7:07 AM

#

hmm ok

#

are you compiling with optimizations at all?

lost schooner Sep 11, 2024, 7:08 AM

#

-O2

#

I'll take it down to -O1

dusky orbit Sep 11, 2024, 7:08 AM

#

does it work with -O0?

#

I know you have UBsan but that doesnt catch everything, maybe you have some ub somewhere?

lost schooner Sep 11, 2024, 7:10 AM

#

works with -O1

#

interesting

#

I'm curious how GCCs analyser works, so I'm going to try that

#

ubsan is off...

#

-O2 with ubsan works

#

so definitely some ub somewhere

lost schooner Sep 11, 2024, 7:17 AM

#

lost schooner I'm curious how GCCs analyser works, so I'm going to try that

doesn't print anything

#

-Wall -Wextra -Wpedantic doesn't show anything interesting

#

In this code:

assert(memoryMap != nullptr);
assert(memoryMap[i] != nullptr);

memoryMap is not nullptr in the first line, but is in the second

#

this doesn't make sense

dusky orbit Sep 11, 2024, 7:29 AM

#

interesting

#

where is that variable allocated?

lost schooner Sep 11, 2024, 7:30 AM

#

not sure. It is a function argument

#

I'm in the process of investigating potential stack corruption

#

I have determined that it is highly likely

#

the stack ends at 0x....4b000, and the current pointer is at 0x...4af10

#

wait no

#

stack grows down

dusky orbit Sep 11, 2024, 7:35 AM

#

what about stack allocated arrays (fixed or variable size)?

lost schooner Sep 11, 2024, 7:36 AM

#

the stack is reasonable

dusky orbit Sep 11, 2024, 7:36 AM

#

do you do anything like that for printf stuff?

lost schooner Sep 11, 2024, 7:36 AM

#

dusky orbit what about stack allocated arrays (fixed or variable size)?

i don't think there is any

#

no

dusky orbit Sep 11, 2024, 7:36 AM

#

hmm

lost schooner Sep 11, 2024, 7:36 AM

#

my printf is fairly light on the stack

#

this is so strange

#

I've read through the assembly a bunch, and it looks like the first assert is disappearing

#

either that or GCC is reordering things

#

source:

#

and the asm:

#

the first assert is gone

celest sierra Sep 11, 2024, 8:00 AM

#

what is assert

lost schooner Sep 11, 2024, 8:00 AM

#

#

since it is C++, it should be the first one

celest sierra Sep 11, 2024, 8:01 AM

#

interesting

lost schooner Sep 11, 2024, 8:02 AM

#

I just pushed the code to here if anyone is interested

dusky orbit Sep 11, 2024, 8:03 AM

#

I did not realize static_cast had so much undefined behaviour in the spec 😔

#

I wonder if there's some assumption being made about memoryMap that leads to the compiler resolving that that inline-if at compile time, and then it removes the void (0)

celest sierra Sep 11, 2024, 8:08 AM

#

yeah you should replace it with assert(expr) do { if (!(expr)) __assert_fail(#expr, __FILE__, __LINE__, __ASSERT_FUNCTION); } while (0)

#

and see if that makes a difference

#

also the #ifdef NDEBUG at the top

#

make sure it isnt being activated lol

lost schooner Sep 11, 2024, 8:34 AM

#

no difference

#

this is quite interesting:

#

the check does occur, but only under a certain circumstance

#

the first memory map entry being read causes the page fault

#

so the check mustn't be happening for that entry for some reason

#

I'm going to try some weirdness to see if I can confirm this

#

huh

#

the check does happen

#

I used a hex editor to replace one of the nops with a int3 in the case where the check happens

#

and I get a breakpoint exception

#

oops. I forgot I had changed the code slightly

#

I cast memoryMap to volatile MemoryMapEntry**

#

and magically the check happens

#

so obviously the compiler is making a false assumption due to some UB

#

yep, I get an assertion failed kernel panic

#

why is it even null

#

and I have noticed that the number of memory map entries is inconsistent on my test device

#

it seems to range from 38 to 42

hardy lake Sep 11, 2024, 11:07 AM

#

lost schooner doesn't print anything

-fanalyze

lost schooner Sep 11, 2024, 11:11 AM

#

I used -fanalyzer I think

#

I'm pretty sure thats what the GCC docs said to use

hardy lake Sep 11, 2024, 11:12 AM

#

There's no way it outputs nothing

delicate blaze Sep 11, 2024, 11:13 AM

#

lost schooner and I have noticed that the number of memory map entries is inconsistent on my t...

just different firmware and memory layout?

hardy lake Sep 11, 2024, 11:13 AM

#

lost schooner and I have noticed that the number of memory map entries is inconsistent on my t...

Makes sense

lost schooner Sep 11, 2024, 8:13 PM

#

hardy lake There's no way it outputs nothing

I'll have another look this afternoon

hardy lake Sep 11, 2024, 8:19 PM

#

Ok

lost schooner Sep 12, 2024, 4:55 AM

#

time to see why GCC analyzer outputs nothing

lost schooner Sep 12, 2024, 4:57 AM

#

hardy lake -fanalyze

#

I added -fdump-analyzer, and it is definitely doing stuff, but nothing is actually printed

#

the dump for my ISR.cpp is 332219 lines long

#

its making vscode use 3GB of RAM

#

I can confirm that the analyser works, just doesn't detect anything

#

so I still have no clue what the issue is

#

I put the stack protector into the -fstack-protector-all mode to see if that would catch anything

#

and it doesn't

lost schooner Sep 12, 2024, 5:15 AM

#

lost schooner it seems to range from 38 to 42

this time it was 46

#

because it is real hardware, I can't use GDB or anything like that

#

my test device doesn't do serial

#

If I disable 2MiB and 1GiB pages, it works

#

I'll try just 2MiB pages

#

my test device's BIOS is so annoying

#

random freezes all the time

#

ok, 2MiB pages work

#

so 1GiB pages are the problem

#

well at least I narrowed it down a bit

#

how about 1GiB but no 2MiB?

#

doesn't work

#

so definitely 1GiB pages

#

not sure why

#

I added a ridiculous number of asserts, and it has gone back to GCC seemingly ignoring them

dusky orbit Sep 12, 2024, 6:53 AM

#

do you check for 1G pages before trying to use them?

#

although iirc that bit is ignored if theyre not supported, at least on intel

celest sierra Sep 12, 2024, 6:56 AM

#

but that would lead to malformed page tables

#FrostyOS