like damn you broke BROKE it
#OBOS (not vibecoded)
vale nymph
flint idol
yeah tbh idk what the fuck happened there
I think it might've been a use-after-free gone wrong
flint idol
yeahhhhh
my allocator doesn't seem to quite like this
specifically vma
it seems to give up and cry on the floor after 106
threads allocated
each with a 0x20000 byte stack
sometimes it gives up
sometimes it deadlocks
or hangs
or the scheduler might just be done with my bs
oh how I hate ipi shootdowns
oh fuck this shit
I genuinely think the test driver binary is cursed
why does only it crash at random times
NO other driver is like that
my kernel is failing this test BADLY
also my vmm (not the rewrite) is not really that thread-safe
ok that could've been fixed
may or may not have been fixed
idk
but otherwise I get a hang
flint idol
good news
it crashes differently
and other good news
the load is more or so evenly balanced out between all 4 cpus
77-78 threads on each cpu's queue
temp = curr_delay/1000;
data.act_x = fixedpt_add(data.act_x, fixedpt_mul(data.vel_x, fixedpt_fromint(temp)));
data.act_y = fixedpt_add(data.act_y, fixedpt_mul(data.vel_y, fixedpt_fromint(temp)));
data.x = fixedpt_fromint(data.act_x);
data.y = fixedpt_fromint(data.act_y);
OBOS_ASSERT(data.x < FRAMEBUFFER_WIDTH);
OBOS_ASSERT(data.y < FRAMEBUFFER_HEIGHT);```this code is buggy
idk what's the problem though
shit
nvm
yeah I haven't any idea
what is happening
oh how I hate fixed point stuff
shit I'm stupid
damn it
why is
'x'
4294123739
flint idol
fixedpt temp_pt = fixedpt_xdiv(fixedpt_fromint(curr_delay), 1000);
data.act_x = fixedpt_add(data.act_x, fixedpt_xmul(data.vel_x, temp_pt));
data.act_y = fixedpt_add(data.act_y, fixedpt_xmul(data.vel_y, temp_pt));
data.x = fixedpt_toint(data.act_x);
data.y = fixedpt_toint(data.act_y);
OBOS_ASSERT(data.x < FRAMEBUFFER_WIDTH);
OBOS_ASSERT(data.y < FRAMEBUFFER_HEIGHT);```ahhhhhhhh
what is wrong with this
interesting
printf("%d %d %d\n", FIXEDPT_BITS, FIXEDPT_FBITS, FIXEDPT_WBITS);
first printf was from the kernel
2nd from the driver
damn it cmake
this better not be a dependency issue
they match now
x is fine now
y
is not
why is y negative
ok so that
is
"fixed"
except now the x and y are the exact same thing each time
as in
they don't change
bruh
why is velocity zero
I've gotten them to change
but they're once again out of bounds
why
is the velocity
that
yeah that's enough for the dat
*day
flint idol
damn it
Assertion failed in function Core_Schedule. File: /home/oberrow/Code/test/obos/src/oboskrnl/scheduler/schedule.c, line 249. chosenThread->status != THREAD_STATUS_RUNNING```ok well it seems as if this thread was from another cpu's queue
bro what is this
flint idol
it almost works
(for one firework)
((it probably wouldn't survive more than one))
flint idol
ok
the thing
is broken
uhhh
the thing was called....
i forogr
ah yes
my delay function
hangs
when blocking
sometimes
causing every thread to eventually hang
the scheduler gives up
after a bit
every thread is cleared from the lists
except the idle thread
and I have no idea why
the ready function does not ever fail
wait maybe I'm dumb
what if the firework finished
nope
at least not entirely finished
stil 61 particles waiting for the (already finished!) delay to finish
feel like this is just a race condition
I swear it feels like something is up with my vmm
firework_data* fw_clone = OBOS_NonPagedPoolAllocator->ZeroAllocate(OBOS_NonPagedPoolAllocator, 1, sizeof(firework_data), nullptr);
memcpy(fw_clone, &data, sizeof(data));
for (int i = 0; i < nParticles; i++)
{
fw_clone->refcount++;
create_thread((void*)particle_handler, (uintptr_t)fw_clone);
}```at the part where it increases fw_clone's refcount
it page faults
and sometimes
it pf's on access of a cpu local struct (that is unpageable)
actually
it page faults on access of some priority list's tail node
it doesn't seem like some sort of concurrency issue
turns out
my scheduler isn't lockless
*isn't supposed to be
omg my poor scheduler
damn it
heap corruption
time to pull out kasan
flint idol
(note: this says "n->next == n" is true)
it found a use after free
from the parent object interestingly
omg
if (--parent->refcount && parent->can_free)
{
OBOS_KernelAllocator->Free(OBOS_KernelAllocator, parent, sizeof(*parent));
parent = nullptr;
}```other than the fact that my scheduler is struggling with this
and the fact that
I hang
when blocking the threads on wait
it works fine, abeit slow
700 threads
flint idol
ok fireworks lesgo
scheduler is holding up well
it's only slow
because of the hundreds of threads which are all ready
and the fact that it takes the particles 2-4 seconds to expire
tbh though the fireworks eventually start to look more like random pixels
than fireworks
I've decided to make it so that a new firework only starts after the old one finishes
that way the system doesn't absolutely die of death
I will be seeing how this turns out on real hw
my favourite part about these fireworks is I get to see as the scheduler speeds up as the particles die
flint idol
I hang
I just hate this
though
I have no idea what is causing it
it was because I was reusing a timer object
without zeroing it
that was the one that blocks the thread on sleep(ms)
.
this is the one that spins
the main differences visually is that the first one seems to give more equal cpu time to the particle threads
whereas the latter is more random
feedback would be appreciated
create_thread((void*)particle_handler, (uintptr_t)fw_clone, THREAD_PRIORITY_NORMAL + mt_random() % THREAD_PRIORITY_URGENT);
using a random priority makes it look cooler
because all the particles aren't started at once
and it's so interesting seeing the difference between different scheduler priorities
on the screen
when particles are of the IDLE priority, it looks so much slower than NORMAL or HIGH
ah night has to be the best time to mess with critical scheduling code
I now test on real hw
and hope that the segfault I got previously is gone
It is
And the kernel works amazingly
@real pecan does my kernel need to receive a gpe from acpi to turn on the fan, or is that (semi-)automatic
I'd assume in acpi mode there is some device you talk to that controls the fan
And you can receive a gpe when the cpu temperature goes past a certain point
real pecan
<@353593948523986945> does my kernel need to receive a gpe from acpi to turn on ...
Depends on the firmware
real pecan
I'd assume in acpi mode there is some device you talk to that controls the fan
There is yeah, its public so u can call it yourself
vale nymph
Gonna make fans on astral be constantly 100%
flint idol
Gonna make fans on astral be constantly 100%
Why, to make it sound like there is always something happening /j
flint idol
Depends on the firmware
Guess mine turns on the fan automatically then, as I heard it turn on running the fireworks test
Today I hope to maybe do some performance measuring to see what functions are the slowest
real pecan
Guess mine turns on the fan automatically then, as I heard it turn on running th...
That would make sense as you don't have any thermal zone or pm drivers to tell them firmware you want to control it
vale nymph
I think on my laptop the fans automatically go to 100% after a while too
flint idol
Oh yeah forgot to mention that the ahci driver is broken
I plan on optimizing the particles to work off timer irqs
flint idol
@real pecan your bootloader doesn't seem to work on my school laptop
Nvm
My usb might be broken
Rather, I have the wrong usb
real pecan
If I had a dollar for every false positive reported by oberrow
flint idol
If I had a dollar for every false positive reported by oberrow
I could have the right one though
It's hard to tell on windows
flint idol
If I had a dollar for every false positive reported by oberrow
You would be the richest man on earth
I'll check if it's the right usb at home
50% sure it is
thick jolt
hi
thick jolt
If I had a dollar for every false positive reported by oberrow
if i had the dollar for every infy is the infy weeeeeeee
flint idol
Good
thick jolt
See this
woahhh
how does the fireworks test work
lol
also whats the point of creating a fireworks test
lmao
congrats tho
OBOS is a real good OS
OBOS compared to nyaux im not your level anymore
😭
inland radish
also whats the point of creating a fireworks test
each particle is one thread
it's supposed to be a scheduler stress test
flint idol
OBOS is a real good OS
Come work with me on it
flint idol
:^)
thick jolt
c++
flint idol
It's C
thick jolt
u sure
flint idol
Ok ty
flint idol
@real pecan the same usb stick boots on my hw
As opposed to the school laptop which doesn't boot
And instead seems to just chainload windows
I made sure secure boot was off
real pecan
I dont have a chainload protocol
flint idol
I know
flint idol
Yes
Because the laptop has one of those splash screen things
And I don't see it reboot
real pecan
Try uefi
real pecan
Uefi hyper does exit on critical errors, but after a keypress
Which would exit back to firmware
So its possible
But idk
flint idol
maybe it PFs or something and the firmware catches that and exits the uefi app
idk though
that sounds weird, but possible
real pecan
Nah thats not a thing.
We can try removing the exit code in hyper after I get back home
Maybe it does exit
real pecan
Yeah thats the strange part
It only exits after a keypress too
So it would at least hang
flint idol
doesn't the firmware have some sorta watchdog
real pecan
Yes, like 5 minutes
flint idol
hmm
real pecan
U have that laptop on you right
real pecan
Oh
flint idol
add as much logs as possible
real pecan
It might also be a buggy boot device picker btw
flint idol
it could be
real pecan
I've had this on one of my laptops
flint idol
because it kinda acted weird
real pecan
U have to choose one specific option
Or it boots into the default one
Or maybe they disabled that for school laptops
Could try limine barebones on it
flint idol
It might also be a buggy boot device picker btw
I think that it still happened when I changed the boot order
unless you mean the thing that actually chooses the boot device, and not the boot menu
real pecan
Yeah
flint idol
Could try limine barebones on it
I'll try that and grub
real pecan
Like from the symptoms ur describing it sounds very very much like it never even gets there
real pecan
I'll try that and grub
Yup
flint idol
anyway, those are all the symptoms I can remember
real pecan
It would make sense to lock down a school laptop tbh
flint idol
I have a plan to make fireworks faster (or at least remove stress from the scheduler)
flint idol
It would make sense to lock down a school laptop tbh
they do for some
some of them they forgot to
real pecan
Lol
flint idol
they also keep the Administrator account open on windows
so I could (theoretically) try and find the password to it
and do some silly stuff
real pecan
Its easy to reset via Linux
flint idol
yeah I know
even booted into a live ubuntu cd once
on the school laptop
now as long as I don't get caught doing this
I will be fine
real pecan
Indeed
flint idol
funny how they lock cmd prompt
but I can just write a very basic script
to get the exact same thing
@echo off
:loop
set /P cmd=%cd%>
%cmd%
goto :loop```with some missing functionality
flint idol
funny how they lock cmd prompt
on some they forget to lock powershell
flint idol
I have a plan to make fireworks faster (or at least remove stress from the sched...
basically (in some non-stress test mode)
I'll make the particle threads start a periodic timer object
which runs every ~16ms
which updates the particle
then after a bit
it wakes the particle thread
which destroys the particle and exits
flint idol
.
@inland radish how does this performance compare to that of your kernel, out of curiosity
inland radish
<@700663172226482254> how does this performance compare to that of your kernel, ...
pretty bad
it seems really stuttery
qemu with tcg
vmware
inland radish
Did you commit your fireworks demo anywhere
flint idol
not yet
inland radish
please do
i wanna see what it looks like in obos
perhaps ill take a look at your scheduler too to see whats wrong with it
flint idol
I'm currently trying to optimize it to not be less stress testy
ofc it will still have the option to stress test
flint idol
perhaps ill take a look at your scheduler too to see whats wrong with it
there is also the video I sent above it that makes sleep blocking
which is slow
inland radish
you really should at least yield to other threads
but tbh even if you dont
they should get the same amount of quantum time
flint idol
they do, unless their priority is different
which I don't think is a problem
or a bad thing
inland radish
in boron all firework threads start at the same priority
they're only boosted by a bit for one quantum by the timer
flint idol
wait shit, I might know what's wrong
the thread's priority is boosted for an entire quantum
of its new priority
say it goes from normal->high
it would go back to normal when its quantum ends
inland radish
that's what i do but there are priority "classes" that the threads cant escape from no matter how hard you try
you can KeReleaseMutex(&mtx, 99999) and the thread waiting on that mutex will never end up in the realtime prio class for example
inland radish
the thread's priority is boosted for an entire quantum
i have a priority base field in my thread struct
a dispatcher function which wakes up a thread usually has a KPRIORITY Increment parameter
this is the priority increment the thread receives when woken up
when the thread is woken up it's emplaced onto the new priority's round robin queue
vale nymph
this talk really makes me want to rewrite my crap scheduler
inland radish
the new priority is calculated by doing min(limit(priority_base), priority_base + increment)
flint idol
I see
flint idol
this talk really makes me want to rewrite my crap scheduler
tbh my scheduler also needs a rewrite
inland radish
where limit(priority_base) is the upper limit of the specific priority class priority_base is on
flint idol
based off this alone
real pecan
tbh my scheduler also needs a rewrite
Stability gone in 1...
flint idol
yup this is the scheduler's fault
with the timer object method it runs a lot faster
@inland radish this is a lot better
as in, faster
real pecan
Whats the idea of this test btw
flint idol
scheduler stress test
real pecan
How does it work
flint idol
it starts a thread for each particle
and there are like hundreds
// Explode it!
// This spawns many, many threads! Cause why not, right?!
int PartCount = Rand() % 100 + 100;```in the words of iprogramincpp
flint idol
bug in my math probably
inland radish
How does it work
hundreds of little threads doing work at the same time, optionally with some sleeps as well
real pecan
Interesting
vale nymph
write an orbital simulation with hundreds of planets where each planet is a thread 
inland radish
i came up with this firework test because i wanted to celebrate the end of 2023
flint idol
I wonder how the performance would differ if I used SSE
inland radish
at the end of 2022 i wrote a fireworks program for nanoshell
nothing extraordinary, everything ran in a single thread
so i was just porting that to boron and then i came up with the idea to split them into separate threads
or at least thats how i remember
maybe i was not directly porting it to boron, not sure
flint idol
what if I just give qemu 128 CPUs, so that I basically depend on the host's scheduler to schedule my threads
inland radish
so i was just porting that to boron and then i came up with the idea to split th...
also i wasnt able to get a stable demo in time for the new year sadly
but i did fix the bugs eventually
flint idol
data->vel_y += (-fixedpt_fromint(10))*temp_pt;
whoops
kernel panic on thread 832
Page fault at 0xffffffff8000619a in kernel-mode while writing page at 0xffffff0000499000, which is unpresent. Error code: 2``` Address Symbol
ffffffff80000d86 oboskrnl!Arch_PageFaultHandler+a6d
ffffffff8000b6bc Unresolved/External
ffffffff8000619a Unresolved/External
ffffffff8005d538 oboskrnl!Mm_VirtualMemoryAlloc+360a
ffffff0000a0753b Unresolved/External
ffffff0000a0a207 Unresolved/External
0000000000000000 End```I hate my vmm
thank god I'm rewriting it
my vmm
flint idol
this one isn't upside down
note it gets slower as the virtual address space gets fragmented, as I cannot free the thread stacks without crashing
to work around this
for now
I will have a free list of thread stacks
so that I can at least reuse thread stacks
instead of allocating a new one each time
real pecan
Now try this test on Linux
flint idol
you mean port it to linux?
real pecan
Yeah
flint idol
that could be something I do soon
short mortar
Ok now I'm suddenly also interested in porting this to my OS...
flint idol
it'd be funny if infy actually wrote uStressTester
flint idol
Ok now I'm suddenly also interested in porting this to my OS...
do it
short mortar
Do you have a link to source?
(Also, I need to study...)
(Random fact: I've suddenly realized that I've entered one of the hardest degrees)
short mortar
Is there an implementation using pthread?
flint idol
doubt
but you can probably still easily port it
just need to change the implementation of CreateThread to use pthread
MmAllocatePool is basically just malloc
then you also need to implement the graphics backend
which is only two functions
PlotPixel and FillScreen
flint idol
I will have a free list of thread stacks
this is going kinda well
except for the fact that there is stack corruption
short mortar
Ok, I think I'm gonna try and port it this weekend
flint idol
ok, gl
short mortar
(Or now)
flint idol
((who needed studying anyway /j))
if (node == free_thread_stacks.head)
free_thread_stacks.head = node->next;
if (node == free_thread_stacks.tail)
free_thread_stacks.tail = node->prev;
if (node->next)
node->next->prev = node->prev;
if (node->prev)
node->prev->next = node->next;
free_thread_stacks.nNodes--;```I need to look at this code really hard
and find out what is wrong
short mortar
I need to be awake in less than 8 hours
flint idol
sleep is a method from the cia to get people to prevent people from becoming batman
don't fall to their tricks
vale nymph
running with smp?
flint idol
ofc
vale nymph
nice
flint idol
I made a change to my timer interface
so it defers execution of the handlers
mainly to spread the handler work over 4 CPUs instead of just the one running the timer irq handler
well more like the timer dpc hanlder
but I disgress
that is my code
currently
@inland radish since you wanted to see the code for my fireworks test
vale nymph
what does DRV_EXPORT do
flint idol
exports a symbol as a driver
my driver interface supports drivers depending on other drivers
flint idol
not used in many places though
only in the test driver as of now
I will now write a test for the allocator (as we all know, I need that desperately)
basically what it does
is allocates a buncha shit
and frees it
multithreaded
flint idol
Cool
flint idol
apparently making firework thingy makes star count go up
interesting
I mean don't get me wrong
firework thingy is cool
oh btw two bugs on real hw with this
a) framebuffer doesn't get cleared properly (if at all)
b) the timer interface seems to be broken (so much for high precision event timer)
from what I could tell, the initial "explodeable" particle is spawned and traverses the screen
then when it dies and spawns in the particles, I assume the timer interface gives up
in the particle handler
I think it could be because I get no scheduler irq
*timer irq
because that's all the timer interface needs
to work
a timer irq (and depending on the platform, a function to reset the timer)
OBOS_NO_KASAN OBOS_NO_UBSAN static void timer_irq(struct irq* i, interrupt_frame* frame, void* userdata, irql oldIrql)
{
OBOS_UNUSED(i);
OBOS_UNUSED(frame);
OBOS_UNUSED(userdata);
OBOS_UNUSED(oldIrql);
#ifdef OBOS_TIMER_IS_DEADLINE
CoreS_ResetTimer();
#endif
if (!work->cpu || LIST_IS_NODE_UNLINKED(dpc_queue, &work->cpu->dpcs, work))
CoreH_InitializeDPC(work, timer_dispatcher, Core_DefaultThreadAffinity);
}```what if this was actually because writing one pixel takes so long
that the kernel lags
I should probably get to making some of my locking situation better
many places use spinlocks, when a mutex might be better
vale nymph
Doesnt your scheduler have priorities
flint idol
I wonder if it'd deadlock with one cpu...
it does...
damn it
damn it
who even has one-core CPUs these days
real pecan
it'd be funny if infy actually wrote `uStressTester`
- Actually exists if you know what I mean
- Probably will have to invent something like that when writing my kernel for testing it
flint idol
1. Actually exists if you know what I mean
2. Probably will have to invent somet...
infy why is the global lock deadlock
oh
wait what
nvm
it deadlocks single-cored
acquiring the global firmware lock
I want to add an FPS counter to the test
I don't really know what to count as a thread though
maybe particles/second?
maybe that is more accurate
inland radish
I want to add an FPS counter to the test
you should rather add a cpu time load counter i guess
thick jolt
infy why is the global lock deadlock
i love deadlocks
im having that issue !
!!!
:)
short mortar
How are you even having them in Rust?
thick jolt
How are you even having them in Rust?
i do not know
lol
move to #1230349543623757845
flint idol
time to deshittify my scheduler more
since the video of the test I sent was using the non stress tested thing
why the fuck is my load balancer so slow
I think since my ready function tries to work balance to the best of its ability
I'll be fine without that slow code path in my scheduler
omg
what the hell
the thread priority booster in the scheduler never runs
flint idol
I fixed that
and I've also decided to add some sorta scheduler profiler
these numbers are in hex
and are gotten through the hpet
and are averaged over each scheduler run between the start of the test
and the end
the work balancer takes 25.6% of the total scheduler time
and these ticks are in 0.01 us ticks
(100000000 hz)
wtf
where is all this scheduler time from
meh whatever
meh whatever
doesn't happen each time
so for some reason
the test decides to spawn hundreds of fireworks
at once
vale nymph
its celebrating
vale nymph
the work balancer takes 25.6% of the total scheduler time
hpet is slow
very slow
theres a reason why qemu is slow as shit in astral
I think it takes the current time to handle the emulation
and it does it a lot
and my gettimeofday() calls directly into the hpet
vale nymph
// TODO: fix me soon (364 days ago)
vernal chasm
(100000000 hz)
Why not in nanoseconds O_O
flint idol
hpet is stoobid
vernal chasm
hpet is stoobid
Did microsoft make it?
flint idol
maybe'
vernal chasm
Cuz they're the only ones who use 100ns
flint idol
Cuz they're the only ones who use 100ns
nah this is just qemu's hpet frequency
real pecan
hpet is slow
btw this is especially true on real hw
real pecan
hpet might need retries and may have insane readback delays
flint idol
it was slow AF
vale nymph
hpet might need retries and may have insane readback delays
retries WTF?:
do you have to handle it or
real pecan
thats why u use TSC
vernal chasm
Solution: use hpet to synchronize the TSC
real pecan
^
real pecan
i had a server that basically had hpet so slow under heavy workloads that linux watchdog would assume TSC was broken because of HPET skew
vernal chasm
yes
real pecan
but in relaity the hpet was broken
flint idol
so how would I calibrate it?
that makes no sense
unless you mena
find its frequency
vernal chasm
Take two samples of the hpet and tsc, and use that to find the rate of the tsc
real pecan
i had a server that basically had hpet so slow under heavy workloads that linux ...
this leads to linux disabling vdso and gettimeofday then becomes a syscall to read hpet, which is absolutely unbelieveably slow
vernal chasm
Other than that you could potentially get the tsc rate directly from cpuid (only intel cpus afaik)
real pecan
your cpu is dog
vernal chasm
ye your cpu is shit 
flint idol
it's haswell
vale nymph
just use the lapic timer to timekeep ez
real pecan
but yeah use lapic
flint idol
does haswell support invariant tsc?
vernal chasm
Afaik any cpu from around 2014 should have invariant tsc
flint idol
Afaik any cpu from around 2014 should have invariant tsc
mine is from 2013
vernal chasm
Could still have it
flint idol
nvm
vernal chasm
You could do what windows does and be shit to anyone with broken tsc and use hpet on those machines xD
flint idol
it doesn't have it
maybe that's why windows is so slow on my machine
becuz no invariant tsc
vale nymph
but yeah use lapic
wouldn't using the lapic for time keeping have a lot of drift?
or whatever its called
real pecan
i think hpet is the next fallback after invariant tsc
its a very complex subsystem on linux so idk
devout niche
i still can't remember what hpet even stands for
high performance entertainment timer or something like that
vernal chasm
high precision... not so much xD
flint idol
ikr
I now need to make timers better
OR
I could drop x86-64 support in favour of m68k support
because on virt m68k timres are simpler
*timers
devout niche
you can just solely support the LAPIC timer in traditional mode
that ought to be sufficient
flint idol
timer_tick CoreS_GetTimerTick()
{
if (!Core_TimerInterfaceInitialized)
return 0;
// if (!cached_divisor)
// cached_divisor = Arch_HPETFrequency/CoreS_TimerFrequency;
// return Arch_HPETAddress->mainCounterValue/cached_divisor;
return curr_timer_tick;
}```well
vernal chasm
I have the solution:
Just run this
timer_value: dq 0
timer_func:
xor rax, rax
.loop:
inc rax
mov [timer_value], rax
jmp .loop
:>
flint idol
obvious solution: have a cpu reserved for the timer interface
vernal chasm
exactly
Apart from that you could also require an external crystal oscillator connected by usb
flint idol
ok tsc time
uint64_t start = rdtsc();
slp(2);
uint64_t end = rdtsc();
uint64_t freq = 2.f/(end-start)*1000;```probably
vale nymph
float in the kernel :(
flint idol
```c
uint64_t start = rdtsc();
slp(2);
uint64_t end = rdtsc();
uint64_t freq = 2...
that math is bad
lemme think really hard
xzczczxczcxz
dfgbdfgdfg5red
unless it isn't
if it takes one ms for the value to go from 5000 -> 8000
what would be the frequency
hmmmmm
shit I forgot math
flint idol
if it takes one ms for the value to go from 5000 -> 8000
these are just example numbers btw
difference is 3000
1ms to go up 3000
1000/(end-start)*1000
maybe?
nope
1/(end-start)*1000
wait that's the same thing
well hz is cycles/sceond or smth
khz is per ms
(end-start)/1000
?
bruh why'd I have to forget math
empty kernel
`1000/(end-start)*1000`
that just cancels to 1/(end-start)
flint idol
yeah ik
vale nymph
Youre sleeping for how much time
flint idol
1ms
vale nymph
end-start is the number of ticks happening in 1 ms, so to find the number of ticks in 1 second is * 1000
vale nymph
Thats like 9th grade physics
flint idol
yeah so tsc works
at least the calibration works
it seems pretty similar to what linux tells me through dmesg
and considering I'm using kvm that makes sense
flint idol
I
hate
timing
so how do I get the tsc of another cpu
tsc counter
without asking it
TODO: Use TSC
Fuck that shit
TODO: Deshittify scheduler
rapid pulsar
On anything recent, the TSC is invariant and should also be identical across cores and even sockets. There's a CPUID bit for it if you want to be super sure.
empty kernel
just came across this: https://github.com/OBOS-dev/obos/blob/master/src/oboskrnl/mm/handler.c#L280-L284
flint idol
just came across this: https://github.com/OBOS-dev/obos/blob/master/src/oboskrnl...
Lol
vale nymph
+1102 -186 lgtm
flint idol
reminds me of this
Page fault at 0xffffff000030d5d0 in kernel-mode while to execute page at 0xffffff000030d5d0, which is present. Error code: 17```
printfs from memprot
0xffffff000030d000 00000003```prot=OBOS_PROTECTION_READ_ONLY|OBOS_PROTECTION_EXECUTABLE
it does indeed change in the memory protection thingy
as in
the pt -has output
flint idol
Fuck that shit
automod doesn't like this
fuck you automod
bruh
so it finally boots
but it uses 5x more memory
(or at least that's what it says?)
flint idol
but why would it use more memory
just because of this change
it doesn't
it's a bug with my vma thingy
I fixed that
(file mapping stopped working)
flint idol
in the driver loader
const char* shstr_table = (const char*)(((uintptr_t)file) + (sectionTable + ehdr->e_shstrndx)->sh_offset);
OBOS_Debug("%p\n", shstr_table);
for (size_t i = 0; i < ehdr->e_shnum; i++)
{
const char* section = shstr_table + sectionTable[i].sh_name;
OBOS_Debug("%d: %p\n", i, section);
if (strcmp(section, OBOS_DRIVER_HEADER_SECTION))
driverHeaderSection = §ionTable[i];
if (driverHeaderSection)
break;
}```from that code
and the pf happens in strcmp
the section table is filled with garbage
and the elf header is barely correct
and by that, I mean it's corrupted, but not corrupted enough for the elf loader to fail
(on disk it's fine)
at offset 0x18 in memory, it gets corrupted
the file
in the page cache
somehow
inland radish
https://github.com/OBOS-dev/obos/commit/8f6e8efd57eb007febde7d78304972fea7ef0bc6
cool
i think it differs greatly from boron
but still
flint idol
i think it differs greatly from boron
I said it was inspired from boron
inland radish
i remember seeing that and thinking the same thing
inland radish
I said it was inspired from boron
i know
flint idol
it only reports to mods
ik
flint idol
in the page cache
the initrd driver isn't reading it properly
inland radish
hmm why are you using a function table
flint idol
and by that, I mean the memory used by the initrd is corrupted
flint idol
hmm why are you using a function table
so that I can support different types of allocators
inland radish
one more thing i notice
your function style is all over the place
you have Mm_AllocateVirtualMemory, SpawnNewExplodeable, create_thread, delay, mt_random, CoreH_ThreadReady
flint idol
I generally try to make globally exported functions follow the naming convention
in kernel-ones specifcally
and static or internal functions just follow whatever my heart desires
internal could mean internal to the driver, or file
inland radish
i see
flint idol
and static or internal functions just follow whatever my heart desires
usually it's snake_case
and struct names follow snake_case
but the members of the struct I just do whatever
the members of structs are inconsistent though
marker
flint idol
the members of structs are inconsistent though
and local variable names
one month ago
I said I would be able to get to userspace by the end of the month
flint idol
and by that, I mean the memory used by the initrd is corrupted
shit
the initrd image itself seems to be broken
flint idol
waiiiit
I was looking at the wrong elf
so I fixed that...
...and now the ahci driver's memory (specifically .text) is corrupted
from 0xffffff0000280000-0xffffff0000282000
at least
flint idol
it's in the elf loader
flint idol
I have fixed a couple bugs
just for it to start remapping memory again
bro 😭
it remaps that same address like 17 times
LES GOOOOOOOOOO
it almost™️
works\
it at least can boot
the fat driver isn't getting loaded because for some reason it can't find the driver header
when printing each section name
weary hound
looks like you got offset 0 for all the strings?
flint idol
<:thinkong:815589724424241152>
oh yeah this uses CoW which I don't quite have implemented
and now that I changed this to a shared mapping
I find out that the pagecache is corrupted
(how convinient, amiright)
kasan tells me nothing
although conviniently, it gets corrupted with the same data each time
hold on...
it's not corrupted
it's not read properly
nvm
I just needa learn to read
the physical memory is only used once
so this has to bge
something overwriting that phys address
and I'm calling it's the ahci driver
nope
weirdly
the same thing seems to happen with other drivers
or well, I should say other mapped files
it was a use-after-free
and I can boot
and load the fat driver
flint idol
it was a use-after-free
specifically, a physical address in the pagecache was being freed when a shared page mapping was freed
but there is one bug that only comes up when rebooting
O_O
(the commit isn't even done)
Mm: Small changes
Misc: Bug fixes```What's left of the vmm rewrite:
- Working set stuff
- Swap stuff
- Reimplementing user->kernel (and vice-versa) copy
flint idol
so I guess I'll start reimplementing the swap backends
obos_status(* swap_resv)(struct swap_device* dev, uintptr_t virt, bool huge_page);
obos_status(* swap_free)(struct swap_device* dev, uintptr_t virt, bool huge_page);
obos_status(*swap_write)(struct swap_device* dev, uintptr_t virt, uintptr_t phys, bool huge_page);
obos_status(* swap_read)(struct swap_device* dev, uintptr_t virt, uintptr_t phys, bool huge_page);```so I guess I can use some sorta hashmap for this
maybe for in-ram swap
for in-ram swap, I will use a hashmap, along with a free list
what if I just use non-paged pool memory
instead of a region
(what could possibly go wrong?)
flint idol
inram swap is implemented
working-set stuff should be easy enough
as in, it should not be any harder than it was last time
and the time before that
I will also start adding the page cache to the standby list
ofc starting tomorrow morning
flint idol
What's left of the vmm rewrite:
- Working set stuff
- Swap stuff
- Reimplementin...
I also need CoW
flint idol
#1064583713184292894 message
marker
I think I might give different options as to what page replacement algorithm I will run
at compile-time
flint idol
so theoretically
I just need to implement swap out/in now
and this will all magically fall into place
flint idol
I just need to implement swap out/in now
swap in from disk + the dirty page writer
well not disk
but whatever
flint idol
I swap in/out now
except it crashes
loading the fat driver
idk why my vmm has beef with the fat driver
flint idol
so I suspect
it's in the page writer thingy
or rather the stuff related to it
since the swap in does not fail
so I think some memprot happens on that address which makes it writeable
but that change isn't reflected
in the standby/dirty lists
wait wth
the page is read only
it's a bug somewhere else
I think I know
flint idol
but that change isn't reflected
my munmap function does not remove nodes from here
so it's possible that it's using
a node that represented a previous, unrelated region
and with kasan on it crashes
delightful
when in doubt rebuild
it fails while swapping in a page
and I have no idea why
all I know is that it couldn't find the page
in the swap device
perhaps
the page is supposed to be in the standby/dirty lists
but isn't for some reason
well it is marked as standby
flint idol
according to qemu-prof.sh
the kernel spends a LOT of time in CoreS_GetNativeTimerTick
and a lot of time in a specific loop in the allocator
flint idol
turns out it's not a good idea to page out non-present pages
so when I fixed that
I can load the fat driver
except the crash with asan still happens
the curse is back\
why is curr.virt=0
because the page is unmapped
I'm gonna be honest
idk what I was expecting
if (!handled && ~ec & PF_EC_PRESENT)
{
// OBOS_Debug("attempting page in at %p\n", addr);
// Try a swap in?
for (volatile bool b = (addr==0xffffff0000200000); b; )
;
page_info curr = {};
MmS_QueryPageInfo(ctx->pt, addr, &curr, nullptr);```so basically
the problem was curr.virt was zero
so the swap driver thingy didn't find a page at that virtual address
so now I need:
- CoW
then I can commit
flint idol
it's only my third rewrite of my vmm
but it is almost done, at last
oh yeah
the m68k port needs to be updated
if I'm being honest idr what I changed
oh yeah
nvm
and clangd has died on me
and other than a couple fixes
the m68k port should be up to date
I just need to implement the signal stuff
(TODO)
other than the fact the it crashes
on the m68k port
it's fine
and it double mmu faults at Arch_PageFaultHandler
at the very beginning
I think it's a stack overflow
it is indeed a stack overflow
it tries to call Arch_GetBootInfoFrom
which in turn tries accessing Arch_BootInfo
and some part of it seems to be paged out
flint idol
it tries to call `Arch_GetBootInfoFrom`
and this seems to be used indirectly somewhere in the pf handler
oh....
inland radish
```c
if (!handled && ~ec & PF_EC_PRESENT)
{
// OBOS_Debug("attempting page i...
for (volatile bool b = (addr==0xffffff0000200000); b; )
;```what is t his
makes no sense
it either doesnt run at all or runs forever
flint idol
```c
for (volatile bool b = (addr==0xffffff0000200000); b; )
;```
Breakpoint
flint idol
turned out my vmm init code was left incomplete
when I was rewriting it
and the only reason this bug was not showing up on x86-64 was because of how vast the address space was
and the reason the m68k port was crashing was because the kernel was overwriting the first couple pages in-use by the kernel
flint idol
as can be seen here
flint idol
uhhh
if (ec & PF_EC_RW)
OBOS_ASSERT(!curr.prot.rw);```this assert is failing on the m68k
port
basically
what it means
is a write fault is happening
on a RW, present page
uintptr_t flags = PT_FLAGS_RESIDENT;
if (!page->prot.rw)
flags |= PT_FLAGS_READONLY;
if (!page->prot.user)
flags |= PT_FLAGS_SUPERVISOR;
if (!page->prot.uc)
flags |= PT_FLAGS_CACHE_COPYBACK;
else
flags |= PT_FLAGS_CACHE_DISABLE;```hmm
the only thing I can think of
is the parent PTs have the RO bit set
hm
-mcpu=68040 is passed to the assembler
ptest:
move.l (4, %sp), %a0
ptest %a0
move.l %mmusr, %d0
rts```function for reference
wait that's the wrong way
I hate at&t syntax
flint idol
wait the page is non-present
fixed
but now paging in is broken (again)
so I fixed that
to get a double mmu fault
..because SP was nullptr?
oh nvm it was a stack overflow
-d int reveals it was a null SP
in CoreS_SwitchToThreadContext
which hints that
a thread was created with no stack
fixed
it was uninitialized memory
m68k:
[ LOG ] Currently at 9208 KiB of committed memory (852 KiB pageable), 844 KiB paged out, and 8356 KiB non-paged.```x86_64:
[ LOG ] Currently at 24376 KiB of committed memory (4456 KiB pageable), 8916 KiB paged out, 19920 KiB non-paged. and 43380 KiB uncommitted.```I don't think these PF counts are good...
I'll also add "hard" and "soft" page fault counts
hard means it swapped in from disk
or needed to read from the fs driver
(for mapped files)
soft means it could use the dirty/standby lists to swap in the page
or the page cache for mapped files
if it's a CoW fault then it's soft
because why not
typedef enum {
// A soft fault is when a swap in could use the dirty or standby lists to do a swap in, the part of the page cache at the file offset
// of the address was already filled in, or the fault was a CoW fault.
SOFT_FAULT,
// A hard fault is when a swap in needed to read from the swap_dev, or a part of the page cache needed to be read from disk
// to satisfy the fault.
HARD_FAULT,
} fault_type;```