I mean it looks correct but also that is probably the longest line in my kernel
#Elysium (Cronus)
honest echo
vocal jackal
can u add logging to vm_map_direct to see what it actually ends up mapping
honest echo
yeah fs
vocal jackal
is it a concious decision to not use the direct map btw?
vocal jackal
why is that?
honest echo
because I'd like to have the option to disable it
vocal jackal
hm
honest echo
its not necessarily rational
just something that I decided on
either way it should not impede this
lgtm
I'm gonna dump the mapped region
and see whats there
honest echo
how so
vocal jackal
the one in < >
honest echo
thats just the region uacpi requested
vocal jackal
i kinda wanted to see what it is u actually end up mapping
vocal jackal
you're using your own bootloader right
honest echo
yes
vocal jackal
how do u fetch the rsdp
honest echo
I did consider the RSDP might be wrong
it looked slightly different but now it looks very similar to limine 
just rewrote it
incase
I mean the previous version worked fine when I was parsing it myself
vocal jackal
can u just dump that address from gdb
vocal jackal
i mean its very easy to check also if u just replace your mapping function with HHDM, if that doesnt work also that must be either wrong address or you overwrote it
honest echo
thats fair
ill try that rn
apparently I cannot set a breakpoint when booting from uefi
like at least not before efi has handed off or whatever
vocal jackal
u should be able to if u disable KVM
vocal jackal
its possible that ur forgetting to flush the tlb properly as well
because your mapping is global
honest echo
I am not
I am super liberal with my tlb shootdowns atm
they are insanely bad for perf
lmao
but at least I am not missing anything atm
vocal jackal
oh well
vocal jackal
what does GDB say
honest echo
yeah doing that
vocal jackal
maybe cast it to acpi_rsdp* so its easy to read
p *(struct acpi_rsdp*)addr or whatever
vocal jackal
it's stored in uefi boot memory so if you're reusing that then that would explain it
u would expect runtimeservices but no
honest echo
I should not be touching that
vocal jackal
maybe print the pointer and contents u get at the time of retrieval
honest echo
time to investigate
vocal jackal
in your bootloader
honest echo
yeah
the pointer is printed and is correct
but the contents I havent
I am going to check a couple things
vocal jackal
yaeh
honest echo
yeah the phys memory is zeroed so the mapping is correct but the memory is fucked
vocal jackal
vocal jackal
yea...
honest echo
idk how I would really set a watchpoint on it
since at the beginning its a physaddr
then its technically present in the hhdm
vocal jackal
set at both?
honest echo
I suppose
actually let me see if the mem is fine at the very beginning of the kernel
that would help narrow the scope
vocal jackal
yeah
vocal jackal
god bless watchpoints
honest echo
at all
real
ok fixed it
that was actually a very good find
at init I was merging buddies that were not free into ones that were free
because at init I find the "max order" of every block
and apparently I was just setting order to that
god
how has this not caused other issues lmao
vocal jackal
lol
honest echo
thanks for the help ❤️
I would have not thought of a watchpoint and would have probably spent hours more on this
vocal jackal
npnp
honest echo
beep boop
frosty oasis
👾🤖
honest echo
I am once again sitting on like 1000-2000 lines of delta without a commit
why do I do this to myself
honest echo
dont execute me for this
honest echo
log packets over serial
lmao
I am going to make it so I can filter by "tag" and log level
which is going to be genuinely useful
vocal jackal
how did u make it into a server?
honest echo
dead simple go app that just reads the logs from tcp and then servers them as server-sent events
vocal jackal
ohh
vocal jackal
reminds me of early nyaux
vocal jackal
usually its because of some sort of an allocator bug always doing a slow path
honest echo
no its uhh
my cpu local accesses for the most part disable interrupts
I am very much going to change that
just that atm its the only way to do it safely
vocal jackal
cli/sti should run pretty fast i think
vocal jackal
well if you're curious u could profile it
honest echo
yeah I have
vocal jackal
its not that hard so thats good
vocal jackal
fair
honest echo
I have other more important stuff to do first
pretty ironic considering what I am working on atm ngl
lol
simple sinew
lmao, thats fun
vocal jackal
pretty ironic considering what I am working on atm ngl
lol i mean
i dont gdt but i have CFI backtraces
my priorities are straight
honest echo
lol
honest echo
ok that took an embarassingly long time
but I am doing this in plain html/js so its a pain to fucking do anything
lol
vocal jackal
would probably be easier to dump in a file and have dmesg read it 
as in just convert to kmsg format
vocal jackal
how do u do symbols and stack traces?
lime vortex
how do u do symbols and stack traces?
speaking of symbols and stack traces, i actually moved Limine to using nm as we mentioned the other day
vocal jackal
yeah i saw the release log
lime vortex
now the symbols resolve fine on non-x86 too, instead of having a bunch of junk symbols
vocal jackal
thats cool
honest echo
how do u do symbols and stack traces?
symbols get loaded as a bootloader module
then I just walk stack frames
sharp surge
why load them as a module instead of putting them into a loadable segment
vocal jackal
symbols get loaded as a bootloader module
do u generate them as an array of structs or
honest echo
why load them as a module instead of putting them into a loadable segment
cuz its easier
honest echo
do u generate them as an array of structs or
pretty much
I just generate a format I can parse at runtime in-kernel
well actually
not parse
just use
honest echo
yeah I mean it'd be neat
but like
bootloader module works just fine
and my kernel can run without the symbols too
tho u wont be able to load kernel modules lol
sharp surge
tho u wont be able to load kernel modules lol
dynsym should already be in a loadable segment by default though?
like with the regular symtab I somewhat understand because you have to do some unconventional linker script stuff to get it loaded, and especially with a custom format symtab, but dynsym is already in memory without any extra tricks
honest echo
yeah I just use the normal symtab
atm
well
not really
I dump symbols with nm
I dont really use kernel modules
and I need to rework them
anyway
vocal jackal
dynsym should already be in a loadable segment by default though?
What's the diff against symtab?
sharp surge
dynsym is exported symbols
and it's what's used by e.g. ld.so
symtab is just debug info
vocal jackal
dynsym is exported symbols
Exported like global?
sharp surge
global symbols with default or protected visibility
vocal jackal
Ah
sharp surge
the symbols that dependent objects can use
you can get it in an executable with -rdynamic or some similar flag
vocal jackal
the symbols that dependent objects can use
Thats actually interesting, so for modules u could make the default visibility hidden and then only export explicitly marked symbols
sharp surge
Yep
And then modules would be normal shared objects instead of relocatable objects like linux modules
vocal jackal
Can't u have that property for objects also?
sharp surge
Meaning you only have to deal with a few very simple relocation types
And none of the relocations will be in read only segments (unless explicitly requested during linking)
sharp surge
Can't u have that property for objects also?
Wdym
vocal jackal
can u have dynsym in objects?
sharp surge
Not really
vocal jackal
sucks
sharp surge
Dynsym is linker output and it only exists for shared objects and executables linked with rdynamic
Objects only have symtab
sharp surge
Yeah idk why they still do it like that when the object format they use now is perfectly capable of actual shared libraries
Like I get the historical reasons but that hasn't been a factor for a very long time
vocal jackal
the only reason i dont like them is because you're forced to use the got stuff
and u cant just have symbols referenced directly
simple sinew
Thats actually interesting, so for modules u could make the default visibility h...
That's what I did in the last iteration of my kernel. Its nice, you can prevent modules from accessing just any symbols they want (from each other or the kernel)
vocal jackal
That's what I did in the last iteration of my kernel. Its nice, you can prevent ...
another approach i've seen is also pretty nice, like what freebsd does and linux has added support for recently
which is have each module specify its namespace
and have other modules import namespaces
so only the respective namespace is looked at for global symbols
honest echo
Thats actually interesting, so for modules u could make the default visibility h...
how would u mark them
vocal jackal
__attribute__((visibility("default")) or whatever
honest echo
ah fair
yeah I need to rework kernel modules
I dont really use em for shit
anyway
simple sinew
so only the respective namespace is looked at for global symbols
mm interesting, I might look into that when I get to implementing modules again.
honest echo
how did u make it into a server?
btw
just realized I qemu can output serial over websockets
lol
well time to do that
this is braindamage but also I need a break from debugging
genuinely why the fuck can it even do that
like its cool and I am about to abuse it
but
vocal jackal
Qemu can do literally anything
honest echo
real
honest echo
I made it so there is no need for a middleman
the website just attempts to connect to the websocket port constantly and so when qemu starts it connects and woop woop
vocal jackal
damn
mystic basalt
neat
honest echo
aight after working on some side projecst I seem to have fixed all the bugs in elysium
dusty hawk
charon rewrite when 😉 💀
honest echo
sike I aint doing that (charon, unfortunately thesis work is unavoidable)
lol
going to continue osdev
simple sinew
lol
honest echo
honest echo
did one for tartarus too
vocal jackal
well what does linux do
honest echo
no clue
jesus christ
linux's is like
somehow more macro aids
they also do a runtime check for the sizes
although that might be optimized out
honest echo
turns out
I got trolled
apparently the assembler can just determine the size of the operation based on the register size of one of the operands just like for mov
so I can just do this
honest echo
magic command that turns my laptop into a hot brick for an hour
honest echo
I love it when I do a couple random innocent changes and now I pagefault in earlyboot
on an instruction fetch
good times
is tartarus still fucking borked somehow
honest echo
well
I have been working on an init system
and so naturally I do the thing I do
and put way too much effort into the dumbest things lmao
but anyway
it now auto generates an interactive init graph
and highlights impossible dependencies
(blue/red here is kind of missleading, they are all okay, only the edges that are dashed are impossible)
I think I might have super overcomplicated things ngl
honest echo
I did not know they had one 😄
I had seen a couple init system type things
and implemented a similar thing
and it became so hard to track what was going on
so I spent a day making an overcomplicated python script to make this
and it sorta helps
one thing I need to do is have a couple big targets that I can depend on instead of depending on every minute thing
honest echo
so gave up on taht
it became such a fucking mess it was unmaintainable
I reverted back to having init stages and still allowing targets to depend on other targets within a stage
its a bit scuffed but works well enough for now
one thing I did inherit from that mess is that I now have a custom lsp that provides completion and diagnostics for target dependencies
which is like
💀
simple sinew
hey thats pretty cool
honest echo
aight I implemented hooks that shouldnt be terrible for performance. I emit the function pointers in a unique section per hook. I also made a python script that automatically generates a linker script that defines start/end symbols because I dont wanna rely on the gnu __start/end ones.
the only thing is that my build process is becoming more complex day by day
lmao
basically no overhead compared to a function pointer call
direct function call would still be more performant but hooks lets me register multiple handlers per hook
extended the lsp to do hooks too
this is actually kinda neat
honest echo
pretty happy with this though
initializing a slab cache for example becomes as simple as this
the hook gets executed from the slab init function
honest echo
did a lot of work on organizing the init stuff
now I am faulting on a nullptr 🙂
fun times
honest echo
I have spent like
15 hours in the last two days
debugging these random faults I get
I cant reproduce them 100% of the time
ther's like 4 different faults ive identified that happens
it only happens using kvm which makes things so fucking shit to debug
its definitely concurrency related
I seem to have fixed them with this
this is in my interrupt handler
though I dont see how this directly fixes the issues I have been having
I definitely see how some issues might arise by allowing soft irqs to nest
but I dont see how it leads to the faults I have been dealing with
😭
so now I am sitting here terrified I havent fixed the root cause
simple sinew
what are the random faults?
honest echo
mostly completely nonsensical page faults
there is one really fun one that kept happening in userspace when mlibc tried to access a kernel space address
and yes I am 1000000% sure it was never passed to userspace via syscalls or anything of that sort
honest echo
thats what came to mind
but fs is intact
gs was zeroed
ds, es are zeroed on every return to userspace
simple sinew
and I imagine you're not touching floating point and vector registers in the kernel?
honest echo
I seem to have *fixed* them with this
the one thing I did think of is that
wait no
yeah idk why this would cause any of that
simple sinew
what is dw_status btw?
honest echo
basically I was allowing deferred work to be handled within nested interrupts
which is probably not ideal at the very least
but I dont see why it'd be fatal like this
and since the issue seems to be very timing related im just worried this changed the timings enough to "fix" it
simple sinew
wrt to the leaked kernel address, I'm wondering if its a page being used twice somewhere (thats not supposed to be).
honest echo
yeah I mean memory corruption is the thing I assumed is the issue
like one thing I found out is if I allocate some memory before some other it "fixes" the issue
which is nonsensical
but god if I know where the memory corruption might be happening
one thing that does come to mind is that if I allow deferred work to be handled in nested interrupts. the likelyhood for them to keep nesting goes way up and I might overrun the stack
that might be the most logical explanation
I really should setup canary pages around the stack
simple sinew
they're worth having
honest echo
actually for now a quick and dirty check is to assert if I am almost off the stack in the handler
and see if that gets triggered
simple sinew
I assume you've tried disabling freeing for various allocators and seeing if the faults still pop up?
honest echo
I assume you've tried disabling freeing for various allocators and seeing if the...
yep
slab/heap. disabling free for the pmm turned out to be slightly problematic
since it gets populated via free
simple sinew
ah lol
honest echo
I should do that though
I am going to try the assert first tho
if it is this I would be so happy
simple sinew
you can also scribble over pages as they're being freed, sometimes that catches things
yeah that would be nice
honest echo
well the assert doesnt trigger
but if I increase the kernel stack size I also dont get the faults
but again that could be because of the difference in allocation
hmm also allocating them with the VM instead of PMM + HHDM seems to fix the thing
though that could be because they are now overwriting other shit, not shit in the HHDM
im going to try some poor mans guard pages
lol
simple sinew
what if you allocate the stack with guard pages on both sides?
honest echo
yeah
I dont have a good impl to do so
but im just going to allocate them somewhere far off in memory far apart
simple sinew
hardcode it in 
honest echo
wow
that is
so I made a static var
just like before the kernel ish
and then I just add some to it for every stack
very scuffed
it..... ends up trying to map this address
what the fuck
well memory corruption it is
oh YES! it happens in tcg
I can watchpoint it
oh
FUCK IM STUPID
it was in decimal
well thats annoying
ig
it works
I suppose that means they are not overunning their stacks
honest echo
you are right tho about poisoning memory
that led to another fault
though STILL it will not happen with tcg on
god this is frustrating
fuck
if I add an assert checking the memory on alloc it apparently causes enough time drift to make the bug not happen
at least it hints towards the pmm at least being intact
honest echo
almost at 20 hours of debugging the same thing
fuck yeah
this totally doesnt suck
I couldve re-written my entire kernel in that time probably
the main thing is I cannot seem to pinpoint what even goes wrong
I know the symptoms but I cannot for the life of me trace any of them back to what actually went wrong
like this one for example
I presume the data in the xsave area got borked at some point
but how to trace it to what did it or even confirm that is what is happening idk
because I cannot run it with tcg so I have super limited means of debugging
honest echo
i dont even remember what I changed
but I think I fixed some tiny bugs
and now its this one that mainly comes up
honest echo
yeah atp I think I am just going to do the thing I have been wanting to do and rewrite the kernel
simple sinew
the sounds of debugging: aaaa
simple sinew
I couldve re-written my entire kernel in that time probably
possibly with new bugs 😔 , leave the rewrites for design changes not bugs.
a bit of an extreme measure but you could do a userspace port of your kernel if its portable. Then you have all your usual tools available to you.
depending on how good your git hygeine is too, you could try git bisect.
honest echo
depending on how good your git hygeine is too, you could try `git bisect`.
it seems this particular thing traces back very far
honest echo
possibly with new bugs 😔 , leave the rewrites for design changes not bugs.
yeah, I think a rewrite is due anyway because I need to redesign some major things
simple sinew
Fair enough, anything in particular?
honest echo
mostly just the way I do like different contexts
like I am not very confident that I manage where I can do different things 100% correct
like in terms of interrupts / deferred work
I need to think it through further before implementing
also the entire kernel was written with a different arch abstraction so its a bit borked atm
honest echo
well spent all of yesterday and the day before rewriting my build system in rust
lol
but its slightly less borked now
honest echo
alright today I will start rewriting cronus
honest echo
cyke
plush mirage
greetings from #showing-off
plush mirage
this matters not
honest echo
though I should start working on cronus soon tho 🙂
plush mirage
work at thy owne pace
honest echo
plan currently is to start over and write a microkernel
plush mirage
plan currently is to start over and write a microkernel
LMAO both Marvin and myself are doing this too
honest echo
but im finishing some tooling first and need to finish aarch64 support for my bootloader
honest echo
LMAO both Marvin and myself are doing this too
lol based
im planning on prolly writing the kernel in C and writing servers & drivers in rust
honest echo
yeah prolly
I write a good amount of rust for normal applications
did a little bit of systems stuff with it
but only dont kernel development in C really
so feels like a decent way to go about it
plush mirage
Then you can surely compile your kernel using #1306765166007816363
honest echo
haha, yeah unfortunately clang only 🙂
I abuse the hell out of clang tooling
re: clang-tidy checks for enforcing atomics
plush mirage
I abuse the hell out of clang tooling
Unless you need compiler internals APIs, you will still be able to compile using Lily-CC because I am planning on implementing a large subset of the GCC and clang extensions.
clang-tidy is just CI checks, you don't use it for actual compilation
honest echo
will check it out forsure 🙂
plush mirage
will check it out forsure 🙂
Oh shit how I need to finish that
honest echo
clang-tidy is just CI checks, you don't use it for actual compilation
yep, though I run it every build
plush mirage
Sounds like it takes a WHILE
honest echo
well I run a very small subset
mainly my custom checks
I use clangs annotate attribute to annotate vars/members that are atomic and then enforce specific functions for accessing them
plush mirage
cool
honest echo
idk its cursed, part of the reason I am interested in using rust
feels like if I need to stretch the language this far maybe I should use something else
plush mirage
I use clangs annotate attribute to annotate vars/members that are atomic and the...
Hmm this is mainly typechecking
That of C is particularly lenient for a strongly-typed language.
I personally use Rust primarily for the borrow checker
plush mirage
Actually since we're both just starting this is a golden opportunity to potentially work together?
Perhaps we could do something like have the same ABI (which is easier because microkernels have less ABI to make in the first place) so we can share work?
honest echo
Sure, we could definitely come up with something 🙂
plush mirage
I don't have any of that designed at the moment.
honest echo
I don't have any of that designed at the moment.
Same, I have only a very rough idea of what I want to do
plush mirage
I do find the idea of Managarm's highly asynchronous I/O appealing
honest echo
I honestly know very little about Managarms internals
plush mirage
So do I