#FrostyOS
1 messages · Page 7 of 1
3 of 4 cpus are stuck
all the issues seem to be related to thread exit
I tried making one set of functions for deleting a thread regardless of what CPU it's running on
but I think that has led to some issues
so I'm going to split it
so there will be one set for exiting the current thread, then another set for terminating a thread on another CPU
I've implemented that
sort of
I just converted most of the existing functions
some nice helpful stack corruption:
CS should not be 0x2b85
similar deal with SS, the interrupt, RIP, and the error code
rflags is also definitely wrong
I increased the stack size from 16k to 64k
and now I get a different page fault
and I just came across the use-after-free in the VMM again
so I think it's kasan time
but I don't want to deal with this right now, so time to play something instead
likely satisfactory
that has also ended up happening tonight...
I'll fix the various bugs eventually, just not really motivated at the moment.
I'm going to attempt to debug this for a bit
there is obviously still some major race conditions somewhere, but I don't know where
double fault followed by a recursive page fault whilst trying to create a stack trace
seems like the stack got deleted mid-execution
seems like the bsp deleted the ap's stack
and then the AP tried to return from a function (x86_64_outb in this case) and it page faulted, then double faulted
there must still be a race condition on thread exit
found it
in work stealing
it should be locking the whole processor state when not running on that processor
but this could cause issues in an interrupt context
so I probably should be disabling interrupts when acquiring a spinlock, and then enabling (if they were before) when releasing
I'll come back to that
but I worked out why it looks like some CPUs might be locking up
because I have per-CPU thread queues
so some CPUs could end up with lots of threads, whilst others could have barely any
so I probably need to implement some sort of balancing
not sure how I would do that efficiently though
I'll come back to that though
got a more important issue
my kernel just ran out of physical memory whilst running the test
and then page faults whilst trying to print the stack trace
the used page count is equal to the total page count
and I checked the numbers, and its close to the 256MiB I give the VM
it definitely could have just ran out of memory
because each thread has a stack which is forced to be paged in immediately since I can't really not do that in kernel mode
since the current stack needs to be valid for a page fault to be handled
I'm going to do a little bit of investigating to make sure the numbers do still line up and that something isn't leaking memory
the total page count is 54408, and the vm region allocator has 53460 in use
so just under 1k pages are still unaccounted for
not 100% sure these are all backed by physical addresses, but they should be
as everything is in kernel mode
which is a bit under 4MiB
the vmm heap is using 727 pages
which still leaves 221 pages, but that is likely all in page tables
so this is likely an actual OOM
I'll increase max ram by 4x to 1GiB
got a general protection fault a bit earlier this time
idk how related this is, but I just discovered that my vm region allocator has no locks
what was I thinking
that could explain many issues
fixed that
now it deadlocks
which I suspect is related to the locks in interrupts that I mentioned before
I'll confirm that though
no it isn't
not those specific locks anyway
both CPUs are likely in an interrupt as the interrupt flag is clear and the TSS has changed to TSS-busy
but the locks are related to the VMM heap it looks like
so idk why there would be allocation in an interrupt
oh I see whats happening on the AP
its not in an interrupt, just part way through thread deletion
so its freeing memory from the VMM
neither are in an interrupt, both are deleting a thread
I'll continue debugging this later
time to continue
the bsp is trying to acquire the heap lock, and the ap is trying to acquire the vmm lock
both of which are already locked
but why is this happening
I have attempted to make this happen, but it isn't going well
so I'm going to play a game instead
I've been thinking about this a bit
and I think spinlocks are a bad lock choice in a lot of places
especially with the kernel being preemptable
using a mutex instead would definitely work
idk if that's the best solution
maybe per-CPU kernel and vmm heaps as well
Mutexes are the way to go where possible
Btw spinlocks are preemptible in RT Linux builds
isnt that potentially quite wasteful? Or is the code that tries to lock aware it wont be able to acquire the lock in its current timeslice and can do something else?
linux rt in general is quite crazy
well they make literally everything preemptible, even RCUs etc
since time slices are enforced very hard
as to how they make sure stuff makes progress im not sure, i think theres some priority boosting going on
yeah if the goal is to be RT I can see how it makes sense
it doesnt seem optimal for general use, but I guess thats not the intent
yeah its useless for consumer hardware
Preemptible by interrupts or by other threads too?
as in, does it block scheduling or not
my own spinlocks are like halfway there because they block scheduling but not interrupts
other threads i think
I'll start on a mutex implementation soon. I'll probably do semaphores at the same time.
I've just committed all my local changes including the debug stuff and temporary fireworks test to the testing branch
I intentionally separated all the debug stuff into one commit so it is easy to confirm later that I have removed it all
the fireworks test will also be removed later
but I thought I should probably commit my changes before starting something sort of new
I've started on a semaphore implementation, but I don't really feel like doing this tonight
Likely won't finish that today, I feel like playing Satisfactory today.
Time to actually do this
the current problem I'm trying to work out is how to return to the current point in execution after blocking execution
but I think I've worked it out
and my semaphore implementation should hopefully be done
I haven't worked out how I'm going to test it though...
my semaphore implementation is also the base for mutexes
I'm tempted to procrastinate by playing satisfactory...
I should now also have a mutex implementation
its just a wrapper around the semaphore one
but I think its satisfactory time...
and that turned into my entire afternoon and evening...
And now its time to start testing my mutexes
I tried enabling them in the heap allocator as a starting point
and get a very early boot recursive page fault
ohhh
gs base isn't set
gs base could definitely be set much earlier
currently it gets set after the vmm is initialised
but it can be set much earlier
it got much further this time
general protection fault though
ah
I see what's happening
initialising a new processor state needs the heap, but the heap mutex needs the gs base (which pointers to the processor state) to be set
the solution is just to have a temporary state on the stack until the actual one is ready
it got to the fireworks test
but it runs much slower now
current issue is that the active thread is deleting its own stack mid-execution
which is very annoying to debug
the faulting instruction is the ret in HPET::GetNSTicks
why not have the bsp allocate per-cpu state for all the cores once it's setup?
that is definitely a better solution
and I've just implemented that
thanks
going back to the bug from before
yep, thanks
When I implemented the returning from an NMI to a halt loop, I chose to clear rsp and rbp, but I'm regretting that now
so I've chose to just set RBP to the current value at the time of building the ISR frame
nice long stack trace:
it appears like the semaphore Wait() is trying to save the thread state so it can block, but the current thread doesn't exist yet
the scheduler is running though
so I just need an extra check
fixed that
next bug is that interrupts are somehow being enabled part way through thread deletion
didn't work that out
but now I've encountered a deadlock
both CPUs are trying to acquire the same spinlock
whatever it is, its in the kernel data
gdb is being very unhelpful and not giving me any clues as to where it might be though
backtrace was actually helpful for once
its the lock for the thread list of the kernel process
I changed it to use a mutex
along with whatever other things use that linked list type
nothing in the scheduler itself uses mutexes, and also should hopefully not allocate outside of init
the current bug is a general protection fault on iretq
I think CS being 0x52c might be an issue...
something is definitely broken
interrupts are enabled
there is still many race conditions
but I don't feel like dealing with them right now
so I think its satisfactory time...
I have thought about just pretending these race conditions don't exist and moving on
but I'll definitely regret that later
That did happen. Not related to this project, but I finished phase 4...
I'll continue debugging this tomorrow
I guess I should continue
and of course first bug is it tries to execute 0 with no stack trace
Idk how to even debug that
the current thread's registers look fine
I've turned KVM off to attempt to make it easier to debug
since qemu's -d flag doesn't work with kvm on
I think using -d int,exec,cpu is not a good idea
Log file is over 9GB and my system just locked up
I managed to get into a TTY and kill qemu, but my system is still struggling
Ah, I only had 8.8GB of storage left...
Maybe I should've checked I had enough storage...
Since btrfs is annoying about usage, my root partition is not in a good state...
Luckily I didn't use up enough that it couldn't even fix itself
Because I've made that mistake before
just spent the last 45 or so minutes fixing my storage situation
log file hit 45GB by the time my kernel started
lmao thats crazy
I like to start logging in places where control transfers happen, e.g. interrupt returns, thread context switches, running dpcs (if you have those), if none of those help maybe check any function pointers before you call them.
or better yet if you know the behaviour before the bug occurs, see if it matches anything else in your kernel (talk yourself through the code if you have to).
Yeah, I think that's a much better option
thanks
the log file is at 48GB, and idk how I'm going to use it when my kernel eventually dies
I'll probably start doing that approach after I have dinner
maybe
I might end up playing satisfactory
lol what
how
.
anyway, back to the various bugs, I'm starting to suspect that stack corruption is happening
because it seems to be a common thing that registers just change their value after returning from an interrupt
I am now logging where every irq returns to
along with all context switches, semaphore wait/signal, and scheduler yields
This is going to take a while for anything to happen
this fireworks test really is good
as annoyed as I am that it is taking ages to fix all the bugs, I'm happy that I'm dealing with a lot of them in one go so they don't cause issues later
its been exactly 2 weeks since I first ported the test
I finally found a bug
took a few attempts of running it
since sometimes it just takes too long
it appears like yielding after blocking the current thread isn't actually working
so this isn't happening tonight
and the debugging must continue
current bug is suspected register state corruption
maybe stack corruption
actually, it seems like the return value of kcalloc is bogus
github copilot found a broken shift in my register save code
which likely explains the bad CS and SS issues I had
this seems to be somewhat consistent
sometimes it dies in the memset, other times when the memory is used immediately after allocation
and it's not isolated to callee-saved registers vs caller-saved
I have to go make dinner now, but I'll continue with this later
and its time to continue
it still seems to be the one bug, and I can't track it down
currently doing a quick ubsan implementation
which is actually looking at how OBOS does it, which is stolen from Sortix, so I've done the same.
I'm getting a type mismatch error in node creation for linked lists, probably alignment related
I'll continue investigating tomorrow maybe
I fixed a couple of other bugs as well
There was one in my lapic timer init where I was using the wrong variable for looking up the divisor value
So it was an out of bounds access, resulting in random garbage being set
Which might explain why the timer felt a bit fast sometimes
The current issue is the kernel effectively deadlocks as both CPUs are idle when they shouldn't be
Seems like maybe threads are disappearing.
I was going to play satisfactory tonight, but I feel really motivated to fix these bugs so I can move on to making a VFS.
I found a bug
I think this screenshot does enough explaining:
fixed that
I just changed Scheduler::RemoveCurrentThread to return what the current thread was
found another bug relating to that one as well
now I'm dealing with some deadlocks
sort of anyway
both cpus are idle
and now a use after free in the VMM
this does not make sense
I played satisfactory for most of the day, so I guess I should probably work on this for a bit
the deadlock is very consistent, so I should probably try work that out first
found the bug
wasn't actually a deadlock
was just missing a check relating to idle threads
some nice stack corruption:
whilst handling an exception of course
and it appears to be trying to execute its own stack
not sure why the error code says non-present
RSP in the screenshot above is garbage
RSP is currently 0xffffc00000081af0
which is the same page as the address it tried to execute
so it would have double faulted if it wasn't present
CR3 is completely sus
most of the registers printed are garbage, I suspect something is returning to what was rbp instead of the actual return address.
What does QEMU log say
I had forgotten that I even had that enabled
time to watch my RAM usage climb as vscode tries to open it
RBP is sus
Is that CR3 sus or normal?
CR3 is normal
Yeah RBP weird
the address in RBP is in the .bss section
LGTM
idk how I'm going to debug this
I tried treating RIP as RBP, but that didn't give any clues
RBP is the address of the BSP's x86_64_Processor class
i just remove the thread from the queue when assigning it
and lock the scheduler when i enter it
they should be getting removed from the queue, and appear to be, but there is some other weirdness going on
maybe you don't lock the scheduler?
the BSP claims to have 3 threads in one of the lists, when it only has 1. The AP claims to have 0 threads in one of the lists, but has 1.
my locking isn't as simple as that
each thread list has its own lock
and each CPU has its own thread lists
my scheduler runs on the lapic timer
but it has a common lock
bc it has a common queue
yeah, it makes sense in your case
I'm not going to debug this tonight
note to self: add logging to all adds/removes to thread lists and fix ProcessorState lock to disable/enable interrupts so it can actually work.
I made it so the locks on the thread lists disable/enable interrupts
haven't done the lock in the processor state yet
something is really messing up the stack
welcome to my personal hell XD
i was stuck on that for a month XD
I've been there
I'm hopeful I'm not going back there
I've changed the stack protector to be strong, and I'll try get some guard pages happening
ubsan isn't catching anything
I don't think kasan is helpful in this case
stack corruption would explain a lot of the issues, but idk where or how
the guard pages is going to be a bit more annoying, so instead I'll just make the stack massive
I normally use 64KiB stacks, but time for 1MiB
got a page fault mid screen clearing, and it actually makes sense
well, the information is consistent anyway
not going to work on this today, playing games instead
I guess I should probably keep trying to debug this
I just inspected all assembly functions that push/pop things from the stack and didn't notice anything wrong
I actually have no idea how or why this is happening
its inconsistent, but its all related to stack data being messed up
I'm going to run this on 1 cpu for a bit to confirm that its definitely SMP related
and it isn't
it has managed to get itself stuck in the idle thread
and I accidentally killed it, but this time it double faulted immediately after switching to a thread
I decided it would probably be helpful to print what the old thread and IP was when printing the new thread when switching
note to self: update fireworks test to latest version
Likely won't be working on this today
I'm on school holidays now though, so I should have a fair bit more time
not sure what you're updating, i haven't changed anything
You changed the velocity of one of the things to scale with screen size
A few weeks ago, but more recently that the version I'm using
Not likely to affect the test for me, but I might as well
Right
I will return to this soon, just playing a lot of space engineers at the moment. Played for almost 14 hours yesterday...
How
9:30am to 11:30pm. Not playing the whole time, had about an hour break at around 6pm, but had the game open and in a world the whole time.
I guess I should return to this
this happened again
I think I know what the problem is
when a thread tries to delete itself, it needs to switch off its current stack. it uses the global kernel stack for that CPU
which is fine if it isn't interrupted
but, as the vmm heap and parts of the VMM use mutexes, interrupts need to be enable
so, if 2 threads are trying to delete at the same time, they corrupt each others stack
A solution would be to have a list of threads waiting to be deleted, and then have a dedicated thread that runs every so often to delete these
in order to do that properly, I should probably setup DPCs
For now, I'll just make the list of threads, and just have a lowest priority kernel thread to deal with them
I could probably use a semaphore to block the thread when the list is empty
the list can probably also be global
actually, a mutex is probably better
just not used as a lock
in the code, I'm using the semaphore class, but restricting the maximum value to 1
that essentially what a mutex is, except the value starts at 1 for a mutex
This should now hopefully be implemented
I was about to say it seems more stable now
but then it page faulted
ohhh
I accidentally made thread exit return...
was supposed to yield
oops
and now it deadlocks
it's waiting for the lock for the list of dead threads
it seems like there might be a case in the kernel panic where it enables interrupts
I couldn't work out what that was, so I just moved on
I reworked the Thread::ExitCurrentThread function a bit so there wasn't so much random locking and other weirdness
seems like now it is trying to run a thread that has been deleted
one of the threads in the dead thread queue is the current thread
but why
and time to continue debugging
I have noticed an issue with the way the thread is deleted
As I'm working on getting this stable on 1 cpu, it isn't an issue yet
but the issue is that the thread is added to the queue and the semaphore is signalled whilst still running on that thread's stack
so with some really bad timing, the thread could be deleted while it is still running
so the original problem still isn't really solved
just a bit different
I think implementing DPCs would solve this, but idk how to implement them
I was reading through the pinned message in #schedulers for how the managarm scheduler works, and I think I know why the stack corruption was happening
in an IRQ, when preemption happens, the old thread gets added to the relevant queue while still running on its stack
so, if another cpu steals it immediately, instant stack corruption
so, the solution is to swap stack
which is also what I should be doing for deleting a thread
I can confirm this is still happening
but idk how
ohh I see what is happening
the force unlock of stdio is causing it
it was stable (and very slow) for much longer this time
until it did that
ubsan was triggered because it was an unaligned access
but that pointer is bogus
I think some stack weirdness might be happening:
or gdb is just being stupid again
gdb is likely being annoying, because my stack trace is fine
so that bad pointer is the head of the waiting thread list for the mutex of the VMM heap
this bug is happening when creating the stack for a new thread
I don't know exactly why it is happening, but I did notice that the thread list data doesn't get cleared when a thread is removed
and the unused fields at the time on insertion also aren't cleared
so that could definitely cause one of the bugs I was seeing earlier
it ran for quite a while this time
then it deadlocked
it's only running on 1 cpu still, so I likely need to either swap the spinlock for a mutex or disable interrupts while it is held
it is the lock for the wAVL tree of mapped regions in the VMM
so, it can be a mutex
another deadlock, this time because of the threads list lock in a process
I think this one is also safe to be a mutex
because threads get removed on a dedicated thread, and I don't think new threads would get created in an interrupt
I'm also turning kvm back on, its too slow without
gdb still partially works with kvm on
about this, they are removed while interrupts are disabled because of the thread list lock
I stopped using kvm as there was some weirdness happening
but tcg really doesn't like this test
its very slow
the amount of threads it creates does not help, along with how big I made the kernel stack...
many threads with 1MiB stacks
took a while this time, but it deadlocked on the dead threads lock
I'm going to implement thread sleeping to speed this up a bit
after I commit my various changes
I'm working on this at the moment, and its a bit more annoying than I thought
I can't swap back to the old stack, but that old stack might have important data
so it can't simply return
I'm working on fixing that
I've just realised that stack switching should be happening in a few more places
After discovering that, I chose to avoid this and played Satisfactory instead.
Got distracted doing other stuff today, I'll probably come back to this tomorrow
I guess I should return to this
I was going to earlier today, but stupid windows decided to reinstall itself
but since I dual boot, when it rebooted I just went into arch, and I'm now avoiding windows
That should now hopefully be implemented
I've done some reordering of where the thread's old state is saved in a couple of the switches as well
still need to do stack switching on yield
and that is implemented
I just realised that it should probably send an EOI on timer tick when it switches...
fixed that
I reduced the stack size to 64K from 1M, and disabled some debug printing
it seems to be getting quite far now
it is still only on 1 core and no kvm, so it's very slow
hundreds of threads with that makes it very slow...
I turned kvm back on, and there is at least 805 threads, but it deadlocked
on the process lock
but interrupts are disabled when it is locked, so maybe something is enabling them again?
I've just realised the problem
the list has a spinlock, with interrupts being disabled/enabled on lock/unlock
but, on insertion and deletion, the linked list methods allocate/free
which means mutex
which can reschedule
I think the best solution is just another intrusive list
I could change it to be another mutex, but I feel like that will cause more issues, and avoiding unnecessary allocation is better
and that should now be implemented
and it appears to work
the test seems to be stable on 1 cpu with kvm on, but quite slow
I could probably speed it up a bit with thread sleeping support
I let it run for little bit, and it died on the lock for the dead threads
it's the same problem as the process thread lock
but this time the solution is going to have to be a bit different
I think I should be able to change it to a mutex though
and that should hopefully be fixed
yeah, no, I really broke something
oh, I see what happened
my current lock solution is not working
I can't use a mutex, because the dying thread can't be interrupted after it switches stack
but a spinlock doesn't work either as the deletion thread could be interrupted as mutexes are used in the deletion process
I think I have a solution though
just need to rework the deletion function to not have the list lock held when the thread is deleted
and that has been implemented
next bug is a null pointer access that was caught by ubsan in thread enumeration in a process
for some reason a thread is null?
which is odd
there is 1104 threads in that list, so I am not manually checking it
I have no idea what happened or how, so I'm just going to pretend it didn't happen and hope it doesn't happen again
I did just find a major slow down though
for some crazy reason, thread deletion from a process was only allowed from its tid, which means enumerating the whole list to find it
when if deletion using the thread itself is done instead, it can just be removed directly
I'll let this run for a little bit. If it survives for a few minutes (mostly just lots of deletions), I'll kill it and start on thread sleeping.
it seems to be stable, and many deletions have happened
and hopefully thread sleeping is implemented
it definitely isn't the best of implementations, but it should work
I have per-CPU lists of sleeping threads, and on a timer interrupt, it iterates through the list and updates the time on all, and adds back threads that are done.
I think I did something wrong, the test just keeps creating new explodeables, but not doing anything on the particles until it eventually OOMs
I'm going to just do a test of the sleep itself
sleep is definitely working on just 1 thread
might be a bit fast, but that isn't an issue for now
it is definitely working, it's just happening very fast
I think its time to run this thing on multiple cores and see what happens
instant null pointer access
its caught by ubsan though
which makes it annoying to debug
because I can't use the register dump and look at the assembly
the vtable on the thread list isn't initialised
which means its constructor was likely not called
turns out memseting the processor state as 0 is not a good idea
and now it deadlocks a bit further into execution
both CPUs are trying to steal from each other
I know why it happens, just working on a solution
turns out the problematic lock was unnecessary
next bug is that both CPUs thread lists are empty
when they definitely shouldn't be
not 100% sure why that was happening, but I did find a release of a lock that shouldn't be happening
got the same bug again, so I obviously didn't fix it
I sadly go back to school tomorrow, so I won't have as much time.
I doubt anyone cares, but I will also progressively have less outside of school free time as the year goes on because of final assignments/exams. Likely not really going to affect this that much for a while though, as my final exams aren't until the end of the year.
Not going to work on this tonight, I didn't end resolving my windows issues the other day, so I'm clean installing. I only really use windows for some games, but I would still like it to be functional and not trying to reinstall itself and fail every few days.
I guess I should continue debugging
I've had a fairly busy week in terms of school, and still will continue to tomorrow, so I've been using a lot of my free time to play games and relax.
I didn't end up doing any debugging, but I got github copilot to review a couple of the main scheduling files a few times, and it found some bugs
Since AI usage in projects seems to be a hot topic in this server at the moment, I guess I'll talk about its usage in this project.
I used to use github copilot for completions all the time, but stopped a little bit ago (probably towards the end of last year when I was doing Frost64 stuff) as it was driving me crazy.
None of the project is vibecoded though, the only AI usage has been under direct supervision with copilot in older code, and more recently for code reviews. I make my own solutions to what it identifies though, not necessarily what it suggests.
There might also be the occasional usage for research, but that is uncommon.
But this project will never have any code that is fully AI generated and any partially generated code (if there is any) is with heavy supervision.
I will attempt to continue debugging
I am very sleep deprived, so hopefully I don't introduce more bugs trying to find/fix existing ones
this still seems to be happening
the kernel process has 165 threads, but none of them are in the active thread lists
this could just be a deadlock, and they are all waiting on each other
the kernel heap semaphore has 20 threads waiting
VMM heap has none
by kernel heap semaphore, I mean mutex. The mutex class is just a wrapper around the semaphore class
none of the other semaphores/mutexes have threads waiting
I didn't end up doing much debugging yesterday, I was too tired to focus.
I decided to work on Frost64 today to avoid race conditions in this, but ended up debugging race conditions in it.
so idk if I will take a break from this and work on frost64, or if I'll continue debugging this
so I ended up working on Frost64, but it's time for me to return to this
so, thread list issues
I got github copilot to scan for bugs, and it found one
when reinserting a previously remove thread, the CPU state pointer in the thread itself was not being updated
there was also a double PickNext
it looks like the test might actually be stable now
just reached round 2 of explodeables (as the test calls them) for the first time
and by round 2, I mean most of the original ones dying, and a new group spawning
All the exits are really slowing things down
highest TID that I've seen in the logs so far is ~4150
nvm ~4500
All particles on the screen are moving though
which means all threads are being activated
by being activated, I mean actually running
I'm just going to let this run for a bit and do some other things
but I think it's stable
it might OOM if threads aren't being deleted fast enough
it's been running for about 20 minutes, and it's still stable
I'm going to try run it on 4 cpus now, as that was only 2
it is very obvious when there is a tlb shootdown
every time one happens, the test stutters
and it died
it ran for a little bit, but then an assert failed when inserting a thread into a list
it looks like the thread might have already been in a list, or its data wasn't cleared when it was removed
I need to track down the address of the thread and have a look
I was about to say it seems stable, but then stack smashing happened
fixed that
I hope
But the test has been running for a bit over 20 minutes and is still stable
KVM isn't on though
so I'll turn it on and let the test run for a bit
it immediately died
this bug is back
ThreadList::pushBack duplicate link: thread=ffffc00002568938 tid=3396 prev=ffffc000024b7618 next=ffffc000024b7758 head=ffffc0000278b668 tail=ffffc000025687f8 count=77
the issue here is that prev and next should be 0
I'm slowly working through the bugs, but they are getting much more annoying to find
I'm using github copilot chat in vscode to help track down the bugs
it finally OOMed!
which might mean the test is actually working
not consistent though
the test appears to be stable on 2 cores, just not 4
This is an issue though
I've made some changes to help improve that, but I probably need to setup an OOM handler that blocks until the list of threads waiting to be deleted is empty.
I've just discovered that my kernel didn't like being given 16G of ram
page faulted in the PMM in init
does this version of limine not map RAM above 4GiB?
I'm going to make sure I'm on the latest version and then try again
updating to 12.x fixed it
I was still on 10.x, and 11.x also had the issue
12.x does mean that I can't get away with using the binary branch anymore though
the test is stable on 2 cores, which I think is good enough for me
I know this probably isn't a good idea, and I'll definitely regret it later, but I've spent the last 2 months on this test
fully stable with KVM off and stable on 2 cores with kvm on is good enough for me
Wow, the testing branch has 44 commits
I really am finally breaking up commits a bit
The next main thing I want to do is a VFS
but I think I'm going to do a couple of small things first
like symbol lookup in stack traces
Note to self for next time: restore RAM amount in qemu run command
and I just fixed that
symbol table time
I'm just going to steal the code from my last rewrite
the tool for converting the nm output is still in the source tree
just need the kernel part
it works:
not sure what the last entry in the stack trace is
probably just some weirdness with how the stack is setup for kernel threads
that did reveal though that there are some symbols before the kernel start
most likely for the AP trampoline
much better:
I guess it's finally VFS time
I haven't done a VFS since my original kernel
the last rewrite died earlier
I started implementing the data structures, but then I realised I still hadn't documented how to build this project
so I have now quickly done that
I'm not sure what I'm going to do for file path lookup yet
I've been looking at Astral for some inspiration for how to implement some of the sun vnode style design since it seems to follow it quite closely
but I'll look at some other OSes and think about it a bit
I'm still not entirely sure about how I'm going to implement the file path lookup
I've decided that each file system will have its own file name lookup system (to an extent anyway)
I'll figure most of it out as I go
but I first need to implement a tempfs so I have some filesystem to test the vfs with
I've started working on the tempfs
read for tempfs vnodes should be done
can't test it until I have write though...
write is basically the same as read except that it can be expanded in size
write should now also be done
I'll test this, and then I'll work on the actual file system part
and it works
After that couple of months of debugging, I'm starting to lose my motivation to work on this.
I'm slowly working on things at the moment.
I'm currently working on child lookup in tempfs vnodes
I ended up actually doing an assignment last night, but I will continue working on this tonight.
tempfs vnode child creation and name lookup is working
now it's time to integrate this into the actual VFS
VFS mount root with tempfs is working
I guess I should implement path resolution
I'm not doing access checks or links at this stage
also not doing working directories yet
path resolution should be done for now
I also didn't handle .. as I can't be bothered right now
creation of directories and folders should also now be done
time to test that, and also path resolution
Isn't tempfs just a fs ram?
yes
Just making sure
and it works:
Do you have the code on github?
Not yet
I'll make a branch and put it on there this afternoon
It's still work in progress, but this is what I have so far: https://github.com/FrostyOS-dev/FrostyOS/tree/vfs-work/kernel/src/fs
I'm making sure to do proper referencing counting for vnodes
I haven't really done it much before, but I feel that the lifetime of vnodes can get quite messy
This should now be implemented
working directories aren't going to be too hard either
This path just confirmed that .. and . are working: /folder/../folder/./test.txt
I guess I'll do working directories now
I had a short KSP distraction, but I've now just implemented working directory support
and after a couple of small bugs, they work
I think I should probably do symlinks soon, and permission checking
also need support for building a path from a vnode (essentially the reverse of looking up a path) for stuff like getcwd
I guess I should actually make a test for them
and they work
I have also merged the vfs-work branch into master
I guess I should implement an initramfs
which is just going to be an uncompressed tar file since that's easy
It works
it isn't actually creating the files/folders yet though, just listing them
Alright, they are actually loading now
after some bugs and procrastination
the main bug was actually a VFS path lookup bug
also, TCG + 1kHz timer is not a good idea
I've changed the timer to only be 500Hz, which is probably still faster than it needs to be
I can confirm that the files are loading correctly:
I guess now I need to work towards ELF loading and system calls
going well so far:
matches the readelf output
I need to implement allocating memory at a specific address in the VMM
so I can actually load programs
I have hopefully implemented that for the VM region allocator
which is the hardest part
Implementing it for the core VMM isn't an issue
I started working on actually loading the file, but I need to implement some sort of clean up system in the loading for when something goes wrong
otherwise there might just be random unused allocated memory regions
It works:
in kernel mode though
I had to do a bit of messing around with making a different process with a different memory region so that it could actually be loaded
and I still need to do the proper cleanup on an error
As for how this was actually printed, I passed the address of the debug_puts function to the test program as an argument
Wait why do you just now have elf loading?
Wait k can't be talking I dont wven have ipc
I've only just implemented a VFS, and I don't do kernel modules.
I've just made something to do this
I have just realised a problem though
loading programs to a different process than the current is a bit of an issue
the pages have to be forced to be paged in as page faults don't work if the pages are not in the current process
which isn't that big of an issue
the problem I was having though is that the pages are getting mapped to the kernel memory, but I realised that I just need to swap page tables prior to loading the ELF file
and I guess when I eventually implement SMAP/SMEP, I'll just need to disable them during the loading process
I'm currently trying to see if I can get this to run in user mode
by this I mean a version of the test program that doesn't actually print anything
it is not going well though
somehow GS has become 0, which has cleared the GS base, causing constant exceptions until the stack overflows
do you proprly execute swapgs where required?
I think so
check every entry / exit between userspace and the kernel along with making sure it only runs on a ring change
because thats how GSbase ends up 0
SMAP can be temp enabked/disabled with stac/clac and SMEP should always be enabled
you could also directly use the HHDM and touch the physicial page instead of that there
this is why i spawn the thread with a kernel function that does the setup while running on that page map so naturally it handles all that nicely. may be worth condidering
same applies to this
Turns out I forgot to swapgs on task switching
It is fine on interrupts, just not on the actual task switch function
It really only should be needed on a drop to userspace function? Iirc I did not have a swapgs on basic task switches, which is somtimes ran on an interrupt and somtimes not
I was thinking about doing something like that, just seems like a bit of extra work that isn't necessary
It’s not really extra work, esp when it simplifies it and it also means the thread that started the spawning can continute right back into userspace and it can waitpid() or whatever right away or do other work
My explanation was terrible, my x86_64_SwitchTask function uses iretq, so it works for both kernel switches after an interrupt and enter userspace. I added a conditional swapgs to it like what happens on regular interrupt return.
Ahhh, mine just works with a normal ret as it only runs from kernel contexts (which is plenty fine)
Makes sense
And the test program is running in user mode (ignore rdi, that's from when I was running the program in kernel mode):
QEMU being stupid:
Maximum number of clients reached
qemu-system-x86_64: warning: dbind: Could not open X display
I can't even take screenshots
something is leaking x sessions
it was discord
I guess that's what happens when I hibernate instead of shutting down for a couple of days
something weird is going on:
CS = 0xb28, SS = 0x5f5b
looks like an ISR frame is getting messed up:
Do you account for some interrupts pushing errors and some not?
Yes
the issue seems to be related to exiting the kernel second stage
probably because it's the main thread
and I'm likely missing some checks related to that
the vscode extension host just crashed, there goes my 50+ hours that discord thought vscode was open for
and it seems like something else is also happening
I tried debugging it a bit more, but then chose to play KSP for 5 hours. I'll continue debugging this after school tomorrow.
the issue has disappeared on tcg, but is still happening on kvm
which is very annoying
since anything with kvm on is annoying to debug
breakpoints and watch points don't work
some -d options such as int also don't work
something is stuffing up the stack
Tcg is slower
Yes, they do
I remember them not working, but maybe I was doing something stupid
ah ok
thanks
I also found a problem that likely isn't breaking anything yet, but will very soon
The TSS RSP0 is the same for all threads on a CPU
which is not good
Yeah, lol
and that should be fixed, now back to the actual problem
that actually fixed the bug
I guess that stack must be used for something else as well
I just found another bug. There are some rare cases where interrupts are not disabled in Scheduler::Yield_Internal (called internal because it is called after swapping stack), which is unlikely to cause issues, but definitely could.
Alright, the ELF loading and other changes/bug fixes have been committed
That is enough for tonight though
I guess next time I should start working on getting system calls happening
System calls are working:
did you add the handler via int 80 too?
No
you prob should
I've chose to just do system calls with the syscall instruction for now
no i mean, when you will port programs
32 bit programs will most likely use int 80
unless you compile them against mlibc
or an edited version of another libc
I want a level of posix compatibility, but I don't necessarily want linux compatibility
I'll be using mlibc
If/when I want 32-bit program support, I'll deal with that problem then
At the moment though, I have no intention of implementing 32-bit support
I'm currently working on an exit system call
but that means implementing proper cleanup functions for all the VMM stuff
this is why I get mad when doing osdev
why is a hhdm address mapped as read only
what is my kernel doing
I'm going to guess that something as mapped something to that physical page as read-only, and somehow stuffed up the HHDM mapping
Are any higher level tables being mapped as RO?
If a higher level table is RO it blocks all writes lower
it is just that 1 page
only HHDM page
What happens if you step through your code when you map that address
See what happens
Pull out GDB
I found the bug, it was because of a bad memcpy when making a user page table
Ah nice
and finally after lots of bugs, I have a functional exit system call
nice one.
I'm currently working on open, close, read, and write system calls
open should now be implemented
I had to implement a bunch of extra stuff first such as fd manager and cwd initialisation/destruction in processes, better process creation, and user copy helpers
close is now also done
I'm not sure if I should copying to a temporary kernel buffer for read/write as the size of the data being transferred could be quite large.
I'll have a look at what other OSes do
I think for read/write I can't really avoid using a temporary kernel buffer
but when I eventually implement mmap, the transfer size issue isn't as much of a problem anymore
read and write should now also be done
time to test them all
wait no, I forgot to implement create support in open
I'm still going to test what I have done so far
none of them are working
open is returning 0 for some reason
when 0 should already be allocated
ohh, forgot to actually intialise the fd manager
open, read and close are working
write is not working for a tty-backed fd
which is odd
I forgot to open the fd.
it works
but I found a tempfs bug
the file size is being page-aligned, which means reading it past the actual end, but not the page-aligned end is allowed
That is a tomorrow problem though
turns out this isn't actually true
but I've also fixed that so new blocks aren't unnecessarily created
that bug is actually a problem, just not in the way I thought
it's an issue with the blocks themselves not the vnode
I chose not to waste an extra 8 bytes storing the exact size of a block in addition to the allocated size
instead I added a check so that the read size was not larger than the file's actual size
I guess I should do mmap, mremap and munmap
no file mapping or shared mappings yet though
mprotect, not mremap
yeah they are pretty easy
They should all hopefully be implemented now. Since I'm lazy, mprotect and munmap require the bounds (addr and length) to be the same as the bounds of the original allocation.
Mostly because splitting memory regions in the VMM is annoying and I don't feel like implementing that right now
and all 3 appear to be working
very unusual for something to work first try, but I guess the system calls are just wrappers around the VMM functions
mmap with a hint address, and both with and without MAP_FIXED responds appropriately for both a used and unused address
I guess that means all 3 system calls are working to the level I want them to be for now
Just remembered that I still didn't do this
That should now be implemented
writing to a tempfs vnode after expanding is causing a page fault, but I think I know why
It's related to how my VMM handles page faults
and now it works
I just needed to add small check to use the kernel VMM instead of the process's one if the process is a user process, and the faulting address is not in the standard user memory region.
I might start working on getting mlibc happening tomorrow
I really don't have many system calls, but I'm starting to get to the point where I need to start making a libc to test the system calls properly
especially with stuff related to kernel headers
I've started working on getting mlibc happening, but all the existing ports seem to use linux headers
which I don't want to do
I'm just going to not mess around with the linux headers and deal with the problems when they occur
well this is not a good start:
/home/frosty515/dev/FrostyOS/FrostyOS/root/data/include/bits/types.h:397:9: error: static assertion failed: "__mlibc_int_fast8 != __INT_FAST8_TYPE__"
397 | __MLIBC_CHECK_TYPE(__mlibc_int_fast8, __INT_FAST8_TYPE__);
| ^~~~~~~~~~~~~~~~~~
/home/frosty515/dev/FrostyOS/FrostyOS/root/data/include/bits/types.h:398:9: error: static assertion failed: "__mlibc_int_fast16 != __INT_FAST16_TYPE__"
398 | __MLIBC_CHECK_TYPE(__mlibc_int_fast16, __INT_FAST16_TYPE__);
| ^~~~~~~~~~~~~~~~~~
/home/frosty515/dev/FrostyOS/FrostyOS/root/data/include/bits/types.h:399:9: error: static assertion failed: "__mlibc_int_fast32 != __INT_FAST32_TYPE__"
399 | __MLIBC_CHECK_TYPE(__mlibc_int_fast32, __INT_FAST32_TYPE__);
| ^~~~~~~~~~~~~~~~~~
/home/frosty515/dev/FrostyOS/FrostyOS/root/data/include/bits/types.h:402:9: error: static assertion failed: "__mlibc_uint_fast8 != __UINT_FAST8_TYPE__"
402 | __MLIBC_CHECK_TYPE(__mlibc_uint_fast8, __UINT_FAST8_TYPE__);
| ^~~~~~~~~~~~~~~~~~
/home/frosty515/dev/FrostyOS/FrostyOS/root/data/include/bits/types.h:403:9: error: static assertion failed: "__mlibc_uint_fast16 != __UINT_FAST16_TYPE__"
403 | __MLIBC_CHECK_TYPE(__mlibc_uint_fast16, __UINT_FAST16_TYPE__);
| ^~~~~~~~~~~~~~~~~~
/home/frosty515/dev/FrostyOS/FrostyOS/root/data/include/bits/types.h:404:9: error: static assertion failed: "__mlibc_uint_fast32 != __UINT_FAST32_TYPE__"
404 | __MLIBC_CHECK_TYPE(__mlibc_uint_fast32, __UINT_FAST32_TYPE__);
| ^~~~~~~~~~~~~~~~~~
My os specific toolchain does not use the same fast int types as mlibc
I guess this is a good opportunity to update gcc and binutils to new versions
and use patch files instead of forked repos
why does the gnu file download site have to be so slow
Pick a different mirror maybe
yeah, I'll look into that. I have no idea where I would find a list of different mirrors. It is more a lack of consistency in speed though.
sometimes it's really fast, it's just really inconsistent
I'm using the official one at the moment, but I'll probably switch to an Australian one
Do you mean http(s)://ftpmirror.gnu.org ?
anyway, binutils has been updated, now I need to update GCC and fix the type issue
yeah
Because that automatically redirects to any other mirror
That’s not one official mirror
That means “use any random mirror”
Oh ok
gcc is currently building
I had a look at the astral patch, and I think the int least issues I was having is because in my gcc patch I was using newlib-stdint.h instead of glibc-stdint.h
I chose to update to binutils 2.45.1 and gcc 15.2.0 as that's what the mlibc guide uses
I was previously using binutils 2.42 and gcc 14.1.1
and it's done
that was quite fast
but it doesn't work
I made a mistake in the LINK_SPEC
it's an easy fix though
I just modified the patch
Alright, its done
that was 8 minutes and 22 seconds
not too bad considering it would take 30-40 minutes on my laptop a few years ago when I was first getting this toolchain working
now it works
it fails at link instead of compile
which is because I haven't actually built mlibc yet
I'm slowly getting mlibc closer to being able to build
It seems like building with posix support, but not linux or glibc is causing issues
lots of things not declared
I didn't really try investigate much though
I'll look into it properly tomorrow
I guess I'll continue trying to work out why mlibc doesn't want to build
It seems like just supporting posix and not glibc and bsd is causing problems
I have no idea what is causing the issues, so I'm essentially copying the tutorial exactly instead of mostly following it
I'm changing my GCC patches to be full unix instead of trying not to be
seems like that was the problem
mlibc is actually building now
it actually compiled
mlibc is trying to use SSE, but my kernel doesn't support that yet
crazy opcode name: punpcklqdq xmm0,xmm1
Seems like I might just need to implement SSE support
x86 opcode or Welsh word. you decide.

I'm very slowly setting up SSE support
It is now enabled, I just need to setup the save/restore
xsave requires 64-byte alignment, but the closest options that my kernel supports are 16-byte with the heap, and 4096-byte (page size) with the VMM.
If I use the heap, I could allocate 48 bytes more that what is required, and use what is required of that as padding
That feels less wasteful than using full pages
I really cannot be bothered implementing 64-byte alignment in the heap
I'm going to do this
You could also make a slab allocator that allows you to configure alignment settings and it takes a set of pages and then splits them up into a bunch of objects based on size and alignment
Then you just are able to spawn instances of allocators of generic sizes / alignment where needed
Yeah, I was thinking about doing that. I just really can't be bothered right now.
(This is similar to just automatically using a padding but it’s just a little more work to do it properly)
Don’t let being lazy stop you from doing things the proper way and then making things worse
It tends to pile up badly and then you have more work to do later
That is true. I've been trying to do things properly for the most part. I just don't really feel like dealing with memory bugs at the moment as it feels like that's all I do.
I know it isn't, but it just feels like it
I likely need to redo my heap to be slab-based at some point
Turns out my heap sometimes isn't outputting 16-byte aligned pointers anyway
and I know why
the HeapSection header is 24 bytes
luckily I use sizeof everywhere in the heap implementation and not magic numbers
so I just add an extra 8 bytes to the header
and FPU save/restore is working
while I was doing that, I also added fs/gs base save/restore
the next thing I need to do is TLS
that was really easy as mlibc does most of the work
I do need to make a proper entry stack layout though
I also need to fix my build system
I've started on this
time to continue
that should hopefully be done now
I'm currently debugging mlibc startup
a general protection fault is occurring when trying to read mlibc::tcb_available_flags in mlibc::this_tid() for potentially the first time
I'll have a closer look with gdb a little bit later
actually, it doesn't make sense
The faulting instruction is: movaps XMMWORD PTR [rbp-0x290],xmm1
RBP minus that offset is a valid, mapped address
the address is not 16-byte aligned though
I wonder if I stuffed up my stack alignment somewhere
yep, I did
my AnonAllocate sysdep is failing now
I wonder if my mapping flags don't match the mlibc ones
yeah, they don't
It now runs without dying, but it doesn't print anything
I don't think it is even reaching main()
something in the VMM is getting messed up
and it is time to continue
I wonder if the page tables are getting messed up
yep, they are
This line is the problem:
raw = (raw & 0x000F'FFFF'FFFF'F001) | ((uint64_t)flags & 0x0000'0F67);
the mask for raw should be 0x000F'FFFF'FFFF'FFFF
I really need to fix my build system
mlibc with debug symbols is too big for my current 10MiB disk image
so I need to manually strip mlibc each time
You could create a larger disk image too to save the trouble
Unless you are constrained for other reasons that aren't immediately clear
Yeah, I'll make it dynamically sized soon
the 10MiB thing is because I'm using the dosfstools and some random mkgpt tool. This is what my cmake target for building the disk image is:
add_custom_target(build_iso
COMMAND ${CMAKE_COMMAND} -E make_directory ${CMAKE_SOURCE_DIR}/iso
COMMAND dd if=/dev/zero of=${CMAKE_SOURCE_DIR}/iso/fat.img bs=1k count=10240
COMMAND mformat -i ${CMAKE_SOURCE_DIR}/iso/fat.img -h 4 -t 80 -s 64 -I 32 ::
COMMAND mmd -i ${CMAKE_SOURCE_DIR}/iso/fat.img ::/EFI
COMMAND mmd -i ${CMAKE_SOURCE_DIR}/iso/fat.img ::/EFI/BOOT
COMMAND mcopy -i ${CMAKE_SOURCE_DIR}/iso/fat.img ${CMAKE_SOURCE_DIR}/dist/boot/EFI/BOOT/BOOTX64.EFI ::/EFI/BOOT
COMMAND mmd -i ${CMAKE_SOURCE_DIR}/iso/fat.img ::/FrostyOS
COMMAND mcopy -i ${CMAKE_SOURCE_DIR}/iso/fat.img ${CMAKE_SOURCE_DIR}/dist/boot/FrostyOS/kernel.elf ::/FrostyOS
COMMAND mcopy -i ${CMAKE_SOURCE_DIR}/iso/fat.img ${CMAKE_SOURCE_DIR}/dist/boot/FrostyOS/ksymbols.map ::/FrostyOS
COMMAND mcopy -i ${CMAKE_SOURCE_DIR}/iso/fat.img ${CMAKE_SOURCE_DIR}/dist/boot/FrostyOS/initramfs.tar ::/FrostyOS
COMMAND mcopy -i ${CMAKE_SOURCE_DIR}/iso/fat.img ${CMAKE_SOURCE_DIR}/dist/boot/limine.conf ::
COMMAND ${CMAKE_SOURCE_DIR}/depend/tools/bin/mkgpt -o ${CMAKE_SOURCE_DIR}/iso/hdimage.bin --image-size 24576 --part ${CMAKE_SOURCE_DIR}/iso/fat.img --type system
BYPRODUCTS ${CMAKE_SOURCE_DIR}/iso/hdimage.bin ${CMAKE_SOURCE_DIR}/iso/fat.img ${CMAKE_SOURCE_DIR}/iso
)
oh wow, you're using cmake to create your image? that's a 7th circle of hell I wouldn't have thought to try, lol.
Not suggesting I do anything any better, but for qemu, I just have a fixed 100MB (eg) drive and I copy the kernel files to it; another setup I use when running on windows is to just mount a folder directly to qemu.
I could use xorriso, but that does mean the fs is ISO 9660 instead of FAT32
I could also use a loopback device, but I'm not sure if that can be user mounted
loopback requires root
God, I created my qemu disks so long ago, I actually don't recall exactly how I did it. I mean I'm pretty sure I just used dd to create the image itself, but then for uefi I would have had to make the filesystem somehow.
Which would have probably been mounted via loopback.
Damn. I'm drawing a complete blank. Sorry man.
(or lady)
Yeah, I thought so. I'll have a look at a few other OSes and see what they do.
Pronouns are in my profile, I'm a man
Just didn't want to assume. Lots of women in os dev and coding nowadays.
fair enough
For some reason mlibc is writing to an address that's in the .text section
My kernel is likely doing something stupid
the ELF is being loaded correctly though
so I wonder if something is returning a garbage address
I was right
mmap is returning a region that overlaps with the .text section
This is why when loading an elf I treat all sections I load as a fixed MMAP so it won’t be allocated later
That's effectively what I do
mmap is just a wrapper around my VMM's AllocatePages function
and the ELF loading uses the AllocatePages function
so there shouldn't be an overlap
looks like the region allocator is at fault
something weird is definitely happening
I suspect with my allocation of a region at a specific address
found a few bugs
still not working properly
mmap is failing though
so I likely still have something wrong
fwiw this is what I do.
fwiw??
for what its worth
Oh ok, thanks
I found the bug
very similar to the other ones
In multiple spots I was forgetting to convert between size in bytes and size in pages
now mlibc initialises
main() is finally reached
finally my hello world program works:
The program was simply:
#include <stdio.h>
int main(void) {
printf("Hello, World!");
return 0;
}
Now I need to actually fix my build system
I use cmake for my kernel, but I need to work out how I can trigger meson
I probably do need to setup package management with xbps or jinx or something
I've worked that out (mostly anyway)
Now I will fix that

