FrostyOS | Operating System Development | Page 8

lost schooner Jun 3, 2026, 9:18 AM

#

And I have done that

#

I'm using xorriso

lost schooner Jun 3, 2026, 10:04 AM

#

and I've finally committed all the stuff I've been doing for the past almost a week

#

I guess I should start implementing more sysdeps

#

I'll do all the get*id ones

#

getpid, getppid, gettid, getuid, geteuid, getgid, getegid

lost schooner Jun 3, 2026, 10:52 AM

#

Now I'll do RTC

#

since I still haven't done it

lost schooner Jun 4, 2026, 8:42 AM

#

Didn't end up doing it last night, so I'm doing it now

lost schooner Jun 6, 2026, 12:31 AM

#

I did do a bit more the other night, and yesterday I was busy all day

#

So, I'll continue now

lost schooner Jun 7, 2026, 12:23 AM

#

I did a little bit yesterday, I'll continue soon.

lost schooner Jun 7, 2026, 2:54 AM

#

I've implemented a GetUnixEpochTime function for getting seconds in a unix timestamp

#

I'll work on the ClockGet sysdep a little bit later

lost schooner Jun 7, 2026, 4:27 AM

#

and that is done

#

currently only CLOCK_REALTIME, CLOCK_MONOTONIC and CLOCK_BOOTTIME are supported

lost schooner Jun 7, 2026, 5:12 AM

#

I've just actually implemented a isatty system call instead of just saying it always is in my sysdep.

lost schooner Jun 7, 2026, 10:20 AM

#

I have successfully implemented directory opening, and reading entries from a directory

lost schooner Jun 7, 2026, 11:11 AM

#

I should probably do fork and execve soon

#

I might do shared mappings and file mapping first though

lost schooner Jun 8, 2026, 9:04 AM

#

I'm working towards shared mapping and file mapping at the moment

lost schooner Jun 9, 2026, 7:11 AM

#

I'm continuing working on that

lost schooner Jun 9, 2026, 11:52 AM

#

Didn't end up doing much, needed to do any assignment

#

but I'll continue tomorrow afternoon

lost schooner Jun 10, 2026, 6:26 AM

#

and that time to continue is now

lost schooner Jun 10, 2026, 6:45 AM

#

first I'm refactoring the VMM a bit

lost schooner Jun 11, 2026, 5:55 AM

#

I've finished the refactoring, now I'll start implementing the shared mapping and file mapping

lost schooner Jun 11, 2026, 6:11 AM

#

I've got distracted trying to speed up the uacpi namespace loading

#

normally it is around 550k ops/s

#

I tried using the vmm heap instead of the kernel heap, and that yields 1M ops/s

#

that shows that my heap is probably the major slow down

#

interestingly, increasing the amount of pages allocated by the heap for adding a new section has a significant performance improvement

lost schooner Jun 11, 2026, 7:31 AM

#

I tried compiling with lto and O3 instead of O2, and it didn't give much of an improvement

#

also, it seems like uacpi as a static library doesn't like lto

#

more specifically ld isn't cooperating

lost schooner Jun 11, 2026, 8:47 AM

#

fixed that

#

turns out I need to tell cmake to use gcc-ar and gcc-ranlib

#

doesn't give much of a performance improvement though

#

it's a 10% boost for many times the compile time

#

so not worth it

#

and O3 is a pain to debug with

#

I get ~1.5M ops/s now

#

the biggest performance boost was allocating new heap sections with a minimum size of 32 pages instead of just 1 page

lost schooner Jun 11, 2026, 11:01 AM

#

shared anonymous mapping is actually quite simple

lost schooner Jun 12, 2026, 8:45 AM

#

I've started working on file mapping

lost schooner Jun 14, 2026, 12:54 AM

#

I didn't do much the other night, and I did other stuff yesterday

#

but, I'm back to working on this today

lost schooner Jun 14, 2026, 11:12 AM

#

I'm still slowing working on that

#

took a couple hour keyboard cleaning break. Side note: the amount of screws in my keyboard is ridiculous. Also took a 4-6 hour satisfactory break (idk how long for certain).

lost schooner Jun 15, 2026, 7:31 AM

#

still working on it

#

very slowly

lost schooner Jun 19, 2026, 4:20 AM

#

It has been a little bit since I gave an update

#

I have not worked on this since when I last sent a message here

#

I've been playing satisfactory most days

lost schooner Jun 21, 2026, 3:36 AM

#

I'm going to try work on this for a bit

#

I don't really remember exactly what I was doing

#

and I very unhelpfully didn't explain anything I was doing here

lost schooner Jun 21, 2026, 10:38 AM

#

I'll try explain what I've done so far

#

the refactoring I mentioned was changing the VMM to use separate AnonMaps and MemoryObjects for each mapping instead of doing everything with MemoryObjects

#

the AnonMap is used for managing the physical pages for that specific mapping

#

and the MemoryObject is used for managing the backing for a mapping

#

which at the moment is always a vnode

#

and idk if there will ever be any other backing types

#

a purely anonymous mapping doesn't have a MemoryObject, and a shared file mapping doesn't have an AnonMap

#

after the refactoring, I changed TempFSVNodes to have a MemoryObject for the whole file that they then map into as the file changes instead of just using anonymous mappings

lost schooner Jun 23, 2026, 5:27 AM

#

I didn't work on this yesterday

#

But I have just implementing allocating a MemoryObject-backed memory region

#

need to fix the freeing and page fault handling

lost schooner Jun 23, 2026, 6:14 AM

#

page fault handling should now be fixed

#

yet to be tested

#

still need to fix free and remap

lost schooner Jun 25, 2026, 10:03 AM

#

I have decided to test mapping

#

just in kernel mode for now

#

won't take much to modify my mmap syscall once I confirm it is working

#

map failed: invalid argument

#

not a good start

#

it maps now

#

Forgot that it should be checking the file size in allocated pages, not bytes

#

but it is currently printing nothing

#

maps, just not printing the file contents

#

looks like it deadlocked

#

fixed that

#

now page fault handling isn't working properly

#

and I fixed that

#

but it still isn't printing the contents

#

I suspect there is some inconsistency with how the Pages are being accessed from the MemoryObject

lost schooner Jun 25, 2026, 10:43 AM

#

It works:

lost schooner Jun 26, 2026, 1:41 AM

#

I probably won't work on this today or tomorrow

#

But when I work on this next, I'll need to make it so mmap for files doesn't fail when an address is provided and MAP_FIXED is not in the flags.

#

Also, my MAP_FIXED behaves like MAP_FIXED_NOREPLACE at the moment, so I should probably also fix that.

#

I could also implement MAP_POPULATE without much extra effort

lost schooner Jun 29, 2026, 10:38 AM

#

I am currently going through a minecraft phase, so it might be a bit before I work on this again

#

I am on school holidays at the moment, so I do have lots of time

lost schooner Jun 30, 2026, 4:08 AM

#

I have decided that I'm going to work on this for a little bit

lost schooner Jun 30, 2026, 4:09 AM

#

lost schooner Also, my `MAP_FIXED` behaves like `MAP_FIXED_NOREPLACE` at the moment, so I shou...

I cannot be bothered dealing with this

#

so I just won't change it until something breaks

lost schooner Jun 30, 2026, 9:40 AM

#

file mapping in system calls should now be implemented

#

haven't tested it yet

lost schooner Jun 30, 2026, 9:59 AM

#

free should now be fixed

#

still need to do remap and actually testing it from userland

lost schooner Jun 30, 2026, 10:26 AM

#

remap should also now be fixed

lost schooner Jun 30, 2026, 11:09 AM

#

my various changes to the VMM for file mapping appear to have broken user-mode processes

lost schooner Jun 30, 2026, 11:34 AM

#

something cursed is happening

#

in this bit of code:

for (uint64_t i = 0; i < pageCount; i++) {
    Anon* anon = map->slots[i];
    if (anon != nullptr) {
        m_pageMapper->RemapPage((uint64_t)virtAddr + i * PAGE_SIZE, prot, user, cacheType);
    } else if (obj != nullptr) {
        Page* page = obj->pages.Find(entry->offset + i);
        if (page != nullptr)
            m_pageMapper->RemapPage((uint64_t)virtAddr + i * PAGE_SIZE, prot, user, cacheType);
    }
}

#

obj is nullptr, but somehow it is getting to the finding of the page

#

and it doesn't page fault

#

I suspect gdb is being stupid

#

turns out the bug was with ELF loading itself, I had just somehow not noticed it before

lost schooner Jun 30, 2026, 12:19 PM

#

just need to fix VMM deletion

#

and that is done

#

file mapping and unmapping is working

#

yet to test remapping

lost schooner Jun 30, 2026, 12:44 PM

#

nice big commit

lost schooner Jul 1, 2026, 1:50 AM

#

I'm currently working on unmapping and remapping segments of memory regions

#

and I just realised that my VMM::FreePages doesn't actually free the VM region

lost schooner Jul 1, 2026, 1:44 PM

#

I worked on that for a bit, got it mostly implemented

#

Then got distracted by Satisfactory for at least 6 hours, likely more.

lost schooner Jul 5, 2026, 11:55 AM

#

my last few days have involved playing far too much Satisfactory

#

so I guess I should probably work on this

#

I'm going to work towards mapping ELF files in directly instead of just mapping an anonymous region and copying the contents

#

not quite sure how that is going to work though

#

I'll have a look at what various other people's OSes do

lost schooner Jul 5, 2026, 12:44 PM

#

I have a rough idea of what to do

#

starting to work on this at originally 10pm, now 10:45pm is not a good idea

#

so this will be a later problem

#

hopefully tomorrow

#

unless I get distracted again...

lost schooner Jul 6, 2026, 2:16 AM

#

lost schooner I'm currently working on unmapping and remapping segments of memory regions

I forgot I was doing this

#

should probably finish that first

#

free should be done

#

I did most of it last time I worked on this

lost schooner Jul 6, 2026, 3:20 AM

#

and it page faulted in VMM::FreePages

#

and I know why

#

so stupid

#

anyway

#

it works now

#

remap time

#

and that should now be implemented

#

I forgot to implement the VmProtect sysdep

lost schooner Jul 6, 2026, 4:01 AM

#

mprotect returned ESPIPE?

#

I have a feeling the errno numbers in my kernel don't match mlibc's

lost schooner Jul 6, 2026, 4:31 AM

#

I was right

#

the actual error is EINVAL

#

fixed that

#

and it works

#

now I can implement file mapping for ELF files

dusky orbit Jul 6, 2026, 5:48 AM

#

cool!

lost schooner Jul 7, 2026, 4:16 AM

#

I worked on this for a little bit this morning

#

but then got distracted

lost schooner Jul 9, 2026, 3:38 AM

#

Been playing a lot of satisfactory over the past few days

#

I'm working on this at the moment though

#

need to finish converting the ELF loader to use file mapping

#

then I need to implement unmapping/remapping of multiple regions at once

lost schooner Jul 9, 2026, 5:17 AM

#

I have finally implemented file mapping for ELF files

#

time to see if it works

#

it appears like it didn't work

#

but there is no error

lost schooner Jul 9, 2026, 5:45 AM

#

it is loading now

#

sort of

#

Looks like there might be some weirdness going on with vnode reading

#

found the bug

#

in vnode reading/writing, I forgot to implement reading offset into a block

#

instead of from the start of a block

#

I hadn't noticed this before because all my tests started from the beginning of the file

#

including initramfs loading

#

and the old ELF loader

lost schooner Jul 9, 2026, 7:04 AM

#

hopefully fixed that

#

now file mapping is failing

#

I can confirm that the vnode problem is fixed

#

idk why file mapping isn't working

#

nvm

#

fixed that

#

elf loading is still returning an error

#

and it is happening in a way where cleanup causes a general protection fault

lost schooner Jul 9, 2026, 12:24 PM

#

got distracted for a few hours

#

but it appears like remap is being weird

#

still investigating with gdb

#

turns out gdb's backtrace was lying to me

#

but I found the bug

#

it was because I was forgetting to round up the page count when allocating remaining anon pages for the difference between file size and memory size

#

and it works.

#

next time I work on this, I'll implement unmapping and remapping across multiple regions

#

and then I think it will be time to merge the vmm-work branch

#

and maybe start porting some things

#

I might switch to dynamic linking first though

#

and I need to do fork/exec

#

this is probably a good time to talk about my plan for API support

#

I intend on having linux source compatibility to some extent, maybe only posix

#

And then I also intend on having a native api

lost schooner Jul 10, 2026, 12:32 AM

#

I should hopefully have implemented unmapping across multiple regions

#

it was actually quite simple

#

just need to test it

#

it appears to work

#

just need to do remap

lost schooner Jul 10, 2026, 2:29 AM

#

sadly had to go outside for a bit

#

but I've just implemented it for remap

#

they both aren't quite working properly

#

found the bug

#

I was forgetting to increment the starting virtual address

#

it works now

#

The last thing to do before I merge the vmm-work branch is fork

#

I might also implement support for shared binaries and dynamic linking to test the file mapping further

#

mlibc does most of the work luckily

lost schooner Jul 10, 2026, 4:16 AM

#

currently working on fork

lost schooner Jul 10, 2026, 4:50 AM

#

forking of the VMM should now hopefully be done

#

just need to finish the fork system call itself

#

then test it

lost schooner Jul 10, 2026, 5:27 AM

#

lost schooner forking of the VMM should now hopefully be done

turns out I forgot about the VM region allocator

#

I'm currently working on file descriptor manager forking

#

so I'll come back to that

lost schooner Jul 10, 2026, 9:05 AM

#

vma forking and fd manager forking is implemented

#

just need to implement forking of the main thread

lost schooner Jul 10, 2026, 9:59 AM

#

and that should be done

#

which means it is time to test this

dusky orbit Jul 10, 2026, 10:02 AM

#

ooh exciting

#

are you testing it with anything in particular?

lost schooner Jul 10, 2026, 10:02 AM

#

just a simple program that runs fork() and prints from the parent and child

#

mlibc likes to take its time to compile after adding a new sysdep

#

and my kernel page faults in VMM forking

#

stupid mistake

#

something weird is going on

#

what I thought was a stupid mistake is something else

#

time to inspect with gdb

#

it seems like something about the way stepping through instructions with kvm on works causes false function calls

#

gdb thinks the function has been called, but all the arguments are optimised out

#

then if I step though a little bit, after a couple of layers of stepping into the last function call in the function, it goes to the actual function call

#

it's a bit weird

lost schooner Jul 10, 2026, 10:19 AM

#

lost schooner something weird is going on

found the bug

#

bad memcpy bounds

#

fork is returning on the parent

#

but not quite working on the child

#

ah

#

COW is not working

#

wait nvm

#

that page isn't writable anyway

#

it is an issue with how registers are saved on system calls

#

they get saved to the current processor to be restored, but never get copied to the thread

#

so if a thread was interrupted by the scheduler mid-syscall, the same problem would occur

#

I've just got lucky so far

#

ohhh

#

I forgot to enable interrupts in the assembly system call handler, so that explains why I haven't noticed this before

#

no page fault this time

#

but still nothing from the child

#

looks like it's a page tables problem

#

both processes have page tables

#

I just realised the bug

#

I forgot to change CR3 in the registers

#

instead of kvm getting stuck because of broken page tables, now there is a general protection fault in x86_64_SwitchTask

#

still on page table loading

#

ohh

#

forgot to convert to physical address...

#

I've found that kvm gets stuck if the page tables contain garbage, and tcg gets stuck if you try set cr3 to a higher half address

#

still not quite working

lost schooner Jul 11, 2026, 12:52 AM

#

I debugged it a little bit further last night and found a few more bugs

#

but it still seems like I forgot to transfer something

#

I'm going to attempt to step through the whole system call and see what happens

#

I found a bug

#

the vma for the new process gets fully initialised, including insertion into its trees

#

when all that needs to happen is setting the region

#

which happens later anyway

#

same error

lost schooner Jul 11, 2026, 7:09 AM

#

I decided I didn't feel like dealing with that

#

and have been playing satisfactory for the past 5 hours....

lost schooner Jul 12, 2026, 6:08 AM

#

probably not going to work on this today

#

and I unfortunately have to go back to school tomorrow, so I won't have as much time

lost schooner Jul 13, 2026, 7:03 AM

#

time to continue debugging this

#

I think I found the bug

#

my system call handler doesn't save cr3

#

and it saves cs and ss wrong

#

which explains one of the other bugs

#

turns out both of those bugs weren't doing anything

#

as the cr3 got changed for the new page tables

#

and the cs/ss saving isn't an issue because it is a ring change

#

so they are already the kernel ones

#

as for the other bug I referred to, I have to set the cs/ss of the new thread when forking

#

which I thought without looking at my syscall code is because they weren't saved

#

because they were 0 in the struct

#

but it was because they weren't being saved at all

#

why is it trying to execute a kernel address from usermode: Page fault in user-mode while trying to execute a present page at address ffffffff800080a3

#

that address is the system call handler

lost schooner Jul 14, 2026, 11:45 AM

#

I don't have much free time this week, so I likely won't work on this much if at all until mid next week.

#

Maybe longer

lost schooner Jul 19, 2026, 12:14 PM

#

I might work on this tomorrow afternoon

#

Had a very busy second half of last week and weekend

lost schooner Jul 20, 2026, 9:10 AM

#

I'm going to attempt to continue debugging this

lost schooner Jul 20, 2026, 9:45 AM

#

I'm currently verifying that everything has been duplicated correctly

#

which it appears to be

#

but I suspect that there is still a COW issue

#

yep, found a bug

#

and it's working!!!

#

but

#

there is an invalid opcode exception not long after

#

I'll come back to that in a moment

lost schooner Jul 20, 2026, 9:53 AM

#

lost schooner yep, found a bug

So, to explain this. I forgot to implement COW for when the AnonMap and Anon both exist.

#

looks like an assertion fails during mlibc exit

#

it appears to be related to frigg logging

#

not completely sure though

lost schooner Jul 21, 2026, 4:45 AM

#

turns out there was still a COW issue

#

I fixed the copying for anonymous regions

#

but not for backed-regions

lost schooner Jul 22, 2026, 9:07 AM

#

Did a bit more debugging

#

I was forgetting to reduce the permissions on read/write pages when forking

#

and another small bug

#

Now both processes finish

#

there is just a general protection fault on VMM deletion

#

and I found the bug

#

fork finally works!!!

lost schooner Jul 22, 2026, 9:31 AM

#

lost schooner I might also implement support for shared binaries and dynamic linking to test t...

I guess it is time to do this

#

for some reason cmake is trying to link the kernel to libgcc 3 times, and libc:

#

I wonder if it is an issue with my gcc patch

#

doesn't seem to be

#

idk why cmake is doing this

lost schooner Jul 22, 2026, 11:45 AM

#

i gave up for tonight after that

#

did not feel like dealing with cmake being a pain

lost schooner Jul 23, 2026, 5:48 AM

#

i modified my gcc and binutils patches a little bit, and changed the configure flags

#

currently doing a full clean build

#

I also needed to fix the exec stack thing with binutils

#

gcc is building at the moment

#

gcc is done, so is the kernel build, currently building mlibc

#

interestingly the full rebuild with the couple of patch and configure changes fixed the weird kernel linking problem

#

which is a bit weird

#

since the patch changes were adding -z max-page-size=4096 to LINK_SPEC in frostyos.h and adding crti.o and crtn.o to extra_parts in libgcc config.host.

#

nvm

#

kernel link failed

#

same problem

#

changed my toolchain file a bit, currently doing a clean build

#

not of binutils or gcc though

#

the issue is cmake being stupid

#

and it worked

#

stupid cmake

lost schooner Jul 23, 2026, 6:50 AM

#

I should hopefully now have implemented support for dynamic executables and interpreter loading

#

no shared library loading yet though

lost schooner Jul 23, 2026, 7:17 AM

#

found a couple of bugs in my elf loader

lost schooner Jul 23, 2026, 7:30 AM

#

lost schooner no shared library loading yet though

nvm, didn't actually need to do this

#

I didn't realise how much mlibc actually does

#

there seems to be some weirdness going on with my vmm when forking though

#

I thought I fixed all those bugs, but obviously not

lost schooner Jul 23, 2026, 9:16 AM

#

seems like enabling interrupts on system calls is causing the problems

#

which means there is a race condition somewhere

#

I tried enabling all of these:

#

and my kernel did not like that:

#

I wonder what mlibc is trying to do

#

oh

#

It's using PROT_NONE

dusky orbit Jul 23, 2026, 9:29 AM

#

ah right

#

it wants to reserve space

#

you'll get a bunch of mmap calls with real protections later

lost schooner Jul 23, 2026, 9:31 AM

#

my VMM doesn't support reserving space at the moment, and also doesn't support the proper behaviour of MAP_FIXED

#

I'll try disabling MLIBC_MMAP_ALLOCATE_DSO for now

#

and see if it works

#

since that seems to be what tells it to reserve a region

#

yeah, it works now

dusky orbit Jul 23, 2026, 9:36 AM

#

nice. It's a common pattern: PROT_NONE to get a contiguous region of address space and then various mmaps() within that region to perform the real operations once you know nothing else will interfere.

lost schooner Jul 23, 2026, 9:40 AM

#

I'll look into implementing support for that

#

since most of the stuff I've doing lately is to improve my VMM

#

I probably need to implement support for replacing regions for execve to work properly when I get around to it

dusky orbit Jul 23, 2026, 9:48 AM

#

yeah fair

lost schooner Jul 23, 2026, 10:33 AM

#

turns out implementing support for replacing region(s) is actually quite simple

lost schooner Jul 23, 2026, 10:55 AM

#

I did realise just before that there is a possible race when replacing a mapping

#

the lazy solution is to just keep the vm region allocator lock held from the first free to after the new region is allocated

#

I could do a bit more messing around to make a cleaner solution, but I can't be bothered

#

turns out my VMM does actually support reserving regions with PROT_NONE

#

I can just remove the check in mmap

#

I didn't intentional add support for it

#

but it does just work

lost schooner Jul 23, 2026, 11:41 AM

#

turns out replacing regions is not working properly

#

I tried debugging it a bit, but I'll continue next time I work on this

#

wow, I've had vscode open for 6 hours, and have spent most of that time working on this

#

did have an about 1 hour break 3ish hours ago

lost schooner Jul 23, 2026, 11:29 PM

#

Just did a bit more debugging, and it is working now

#

#

dynamic linking & region reserving is working

#

Next is probably execve

#FrostyOS