hm
#Zinnia
near tartan
the problem is that i want the read of thread to be atomic
otherwise i'd just return gs:0 and cast the struct
but that's not atomic afaict
near tartan
huh
hoary cave
why bother with gs:0 then
if you can just get a pointer to per cpu data
and grab the arc out of it yourself
near tartan
well that's not atomic, no?
hoary cave
i guess
near tartan
this is what i'm working on rn
hoary cave
O_O
near tartan
get_per_cpu works, but get_thread doesn't (because trash code)
hoary cave
yeah i guess you could just Arc::into_raw to leak it and then Arc::from_raw
don't forget to drop(Arc::from_raw(...)) when replacing the current thread pointer :^)
near tartan
duh
okay fuck
i need a better way of setting up percpu data
uacpi wants to read the current thread id, but there is no thread running at that point
i do:
- uacpi init
- cpu init using madt
hoary cave
you should have a "main kernel thread"
hoary cave
realistically it's the first thing you should do 
and current thread shouldn't be an Option tbh
near tartan
i guess
hoary cave
you can't not be running any thread
you always are running within the context of a thread
near tartan
how do i initialize it though
near tartan
i have no scheduler running :P
hoary cave
you can have a static percpu
near tartan
maybe some section hack can do?
somber solar
uacpi wants to read the current thread id, but there is no thread running at tha...
just use whatever your main kernel thread id will be
or just random value
near tartan
that might work for the time being, but in the future this won't be sufficient
somber solar
just add a check if the scheduler is initialised
^
near tartan
i would rather just have the scheduler with a static thread to start with
hoary cave
just initialize your bsp idle thread from a static variable and when you initialize everything repurpose _start as it's idle loop
that would be bsp's current thread when you initialize the bsp per cpu data
(you can also do that from a static)
ideally you'd do that before anything else
i.e first thing after reaching _start
near tartan
hm then i need to rip a lot of this code out
woe
i think it would be better if we can look at this together tonight
i'm probably going to fuck this up in some way lol
like
ideally i would like the scheduler to be ready ASAP
hoary cave
i mean i can try to help but i'm not sure i'm more capable of doing this than yourself
near tartan
2 > 1
hoary cave
yep
near tartan
the scheduler should be setup statically, if that's even possible
maybe some const fn hacks
hoary cave
oh shit, arc allocates memory
near tartan
yes
hoary cave
no good
hoary cave
but you definitely want to refcount your threads
that could work but
that requires a memory allocator to be up
near tartan
well menix initializes in stages
early, memory and init
everything relevant happens in init
hoary cave
ideally the per cpu data would be initialized early...
near tartan
we can set it up after the memory stage
btw it would be very funny if we could do something like managarm stages
maybe @marble salmon can explain how it works
near tartan
i have tried
hoary cave
but i'm not sure if it fits well into the rust type system
with like
borrow checking and all
thought i think there aren't any cyclic references
so it should be fine
near tartan
@hoary cave do you think we should use an Arc for the page table as well?
hoary cave
it would be nice
near tartan
cuz currently it just does this
hoary cave
to be able to refcount the page tables
near tartan
which is obviously a violation
hoary cave
the small problem with using an Arc for the [PageTableEntry; 512] thing is that it is already 4096 bytes big
at least on x86_64
and you doing Arc<[...; 4096], PageAlloc> would alloc two pages per page table
since it needs to allocate metadata too
which.. sucks
near tartan
i think there needs to be a better solution for phys mem alloc
hoary cave
so i suggest Arc'ing the wrapper around page tables
near tartan
i don't think we should rely on an allocator crate tbh
hoary cave
and deallocating the actual page table page in Drop
marble salmon
why do you need to refcount page tables?
near tartan
i guess so you don't drop a thread's page table when it's still being worked on
near tartan
@hoary cave i ended up changing all uses of VirtAddr and PhysAddr to be like what maestro does
i.e. transparent structs with a usize field
that way you can't accidentally use a physaddr in place of a virtual one
(i found 2 places where this happened)
i think it's time to do the scheduler
somber solar
what are you planning for it?
near tartan
nothing too fancy yet
round robin probably
a CFS would be nice
i do have to nuke/redo the per cpu init code
because it won't work as i intended
it basically combined cpu setup + tcb setup which is suboptimal
somber solar
a CFS would be nice
its alternative BFS looks interesting as well
near tartan
if i want to be very fancy, i might even abstract the algorithm away behind a cargo feature
somber solar
it basically combined cpu setup + tcb setup which is suboptimal
tcb?
somber solar
ah
near tartan
idk how to design the api properly
oh damn i also need to do the website at some point 💀
somber solar
for menix?
near tartan
i'm a brainlet
hoary cave
same
near tartan
i changed the kernel boot order so instead of
limine -> arch init -> main
it does
arch init -> main -> limine -> main
but i forgot that limine calls the kernel itself
so it kept initializing itself
somber solar
<@796462670995390524> i may be cooking
what's "present" for
somber solar
push
near tartan
what's "present" for
i had the deranged idea of having some way to "remove" cpus
but for now it's unused
i guess i can add it later
somber solar
lol
near tartan
i mean qemu supports cpu hotplug
somber solar
wha
somber solar
I guess that could be useful for multi socket servers
near tartan
most machines don't allow you to do this (because coolers and so on), but it's still funny
pine rock
I guess that could be useful for multi socket servers
its for ec2 and azure
you can scale an instance up and down as demand changes
until one guy caches the result of std::thread::hardware_concurrency() at which point you can now not scale
full topaz
i mean qemu supports cpu hotplug
huh?
has there been any real hardware that supports cpu hotplug?
near tartan
s390 
near tartan
yes
full topaz
s390 <:trl:829316641191886888>
does linux support it?
near tartan
yes
full topaz
(or windows server for that matter)
full topaz
windows server barely functions on arm
fair point
pine rock
Windows server can do hotplug but not unplug
And it produces some really fucky thread groups when you do hotplug
full topaz
Windows server can do hotplug but not unplug
How do you even know that?
pine rock
because they say it in their whitepaper on hotplug support?
full topaz
because they say it in their whitepaper on hotplug support?
They have a whitepaper on cpu hotplug support?
pine rock
mostly marketing fluf but yes
also you can literally download windows server and run it
https://www.microsoft.com/en-us/evalcenter/download-windows-server-2025 idk what more you could want really
full topaz
also you can literally download windows server and run it
Why would i do that if i could download linux instead?
somber solar
None
near tartan
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
i think i finally figured out why i keep getting page faults
i swap away the gsbase 🗿
but not back
how did i not see this
near tartan
OH MY FUCKING GOD
i swapped the pop and push registers
they pop/push in the same order
instead of opposite order
wow so cool now i don't get random allocation failures
hell yes
can confirm that everything is working as intended now
even runs flawlessly on all my laptops
so the only thing left to do is to switch to my own allocator
correction: almost all laptops
but that's fixable
stupid ass hp envy
full topaz
because they say it in their whitepaper on hotplug support?
btw, can you send a link for that whitepaper pls?
somber solar
yes
near tartan
there may be zeroed memory
somber solar
if it's zeroed memory, what's the point of iterating over it
it's basically "allocate more, read less, iterate over more"
near tartan
null tag could be done like that
also don't forget that the dynamic segment may contain other things than tags (although that's probably not correct)
somber solar
it's basically "allocate more, read less, iterate over more"
^ but it's just goes over the memory that was returned from kmalloc and not touched by read()
so I don't see how it's related to elf contents
near tartan
that alloc should be a zeroing alloc
somber solar
it still doesn't make any sense, because no case would match in that switch to zeroed memory
near tartan
ah good catch, there should be a break in there if a DT_NULL is hit
somber solar
wait if it's just zero after p_filesz and you know it'll be matched, why even allocate and read memsz
near tartan
spec shenanigans, the other PT headers like PT_LOAD do the same
somber solar
I am become confucius, master at being confused
near tartan
like, this is how the .bss works
somber solar
yes but the allocated memory isn't used anywhere
near tartan
filesz 0, but memsz !0
somber solar
somber solar
filesz 0, but memsz !0
if filesz is 0, nothing is read() and nothing is done
just memory and cycles wasted
somber solar
learning menix
near tartan
cyan nexus
same here lol
near tartan
finally
near tartan
@hoary cave progress
i made the decision to always ignore memory below 64KiB so i can do the trampoline shit without a worry in the world
grave peak
What did you change?
near tartan
allocators
have my own now instead of using a crate
but it's still dying a little bit
near tartan
yes
grave peak
L but lfg
grave peak
Its a huge project that's true
near tartan
i fixed the bug...
allocator works now
i accidentally didn't update the phys base of a region so it gave out the metadata as usable memory 🗿
grave peak
No struct page skill issue
grave peak
Letsgo
near tartan
need to think of a way how to best do virtual memory allocations
right now i just have bump allocation
but that doesn't sound like a great idea
ionic lava
ok. why that environment is always "unsound"? I've already used to see this print, looking over screenshots here. Still have no idea what does it mean though.
near tartan
unsound means something like insecure#
like, something has messed up the state in a way that's not supposed to happen
ionic lava
I see, thanks. Is it rust related in this context or more general?
near tartan
just in general, but it is to say "something in unsafe code has caused wrong behavior"
@grave peak ooooooof
pain
to be fair, there's a shit ton of things still left to optimize here
i have to hook up my fast memcpy, the regular malloc impl just uses the physical allocator + hhdm (no caching whatsoever)
ionic lava
From 5,239,043 to 235,000. Was that number achieved with rust btw?
near tartan
5M was in C
235k is rust
but it's not because of rust
the entire code base is different
the allocator i have now is missing a lot
i'm sure once i add my slab allocation back in, this score is going to get much better
ionic lava
Genuinely interested to see the results between functionally on pair implementations in C and rust.
near tartan
i've replaced the rust mem* functions with my own (primitive) ones, and it's already a bit faster
the most part is still going to be the allocator
near tartan
bruh it triple faults on real hw
i think something is not mapped
I don't even get to the fbcon
i gotta check the mapping again
grave peak
<@353593948523986945> ooooooof
Old nyaux perf
near tartan
okay nevermind
@grave peak kernel crashes because of something uacpi does
idk yet what exactly
but if i turn acpi off it doesn't triple fault
grave peak
99% your bug
near tartan
i didnt say it's an uacpi bug
just said that uacpi is doing something that's not nice to my interfaces
grave peak
real and true
near tartan
it's probably a virt memory bug
fucking hell
wish i could debug with serial
i wonder what the issue is
yooo lets go i got access to a server with com1
lfg
near tartan
@grave peak lol?
[ 0.000000] acpi: Initializing...
[ 0.004574] acpi: FACS 0x00000000A890D000 00000040
[ 0.097220] error: acpi: unable to attach (PCI_Config) operation region \_SB_.PC00.PMC1.PMCB: unimplemented
[ 0.111672] error: acpi: aborting method invocation due to previous error: unimplemented
[ 0.123506] error: acpi: #0 in \PMB1()
[ 0.128822] error: acpi: Op 0x5B80 ('OpRegionOp'): invalid argument type: Uninitialized, expected a Operand := TermArg => Integer
[ 0.146356] error: acpi: aborting table load due to previous error: incompatible AML object type
[ 0.159183] error: acpi: #0 in \()
[ 0.163971] error: acpi: failed to load 'DSDT' (OEM ID 'LENOVO' OEM Table ID 'THINKSYS'): incompatible AML object type
[ 0.180268] error: Kernel panic - Environment is unsound!
[ 0.187921] error: at kernel/src/generic/platform/acpi/uacpi.rs:63:56
[ 0.196679] error: "called `Result::unwrap()` on an `Err` value: Utf8Error { valid_up_to: 26, error_len: Some(1) }"
whats going on here
near tartan
why is your print function trying to print invalid data tho
grave peak
idk, more info needed
grave peak
whats the string that it prints
near tartan
not a valid c string apparently
grave peak
doubt
near tartan
could the allocator be to blame for this?
grave peak
if you give me a backtrace i could tell you
near tartan
the error means that after 26 characters it encountered a non utf-8 char
don't have that yet
that test machine is super flaky
but yes, it's probably my allocator
woe
grave peak
lol
near tartan
i need to put more asserts in my allocator
(and probably actually do slab allocation)
cyan nexus
pine rock
(and probably actually do slab allocation)
could i interest you in tlsf allocation instead
pine rock
two level segregated fit allocation
and its got very good fragmentation, and has O(1) allocation and free
https://ricefields.me/2024/04/20/tlsf-allocator.html this is about a zig version but im sure its not too hard to translate to crust
its what vulkanmemoryallocator and d3d12ma use as their internal allocator, so plenty of prior art
near tartan
and its got very good fragmentation, and has O(1) allocation and free
nice
marble salmon
<https://ricefields.me/2024/04/20/tlsf-allocator.html> this is about a zig versi...
This is copied from jemalloc but worse 
It's also not O(1)
Or am i missing something
pine rock
its O(1) if you use the good fit algorithm rather than exhaustive best fit
its a couple of bitmap tests and then either segmenting the null block or another free block
if you do best fit you walk the freelist for 1 or 2 buckets
http://www.gii.upv.es/tlsf/files/papers/ecrts04_tlsf.pdf heres the original paper
near tartan
@grave peakup to 1mil again
oh wow yea this really is because i have no real malloc
using buddy allocation for that is bad it seems
grave peak
Nice
near tartan
i expect this to go up a lot
i ran with 128M
if i put e.g. 4G ram in qemu then performance tanks
near tartan
hell yea i just got access to a two socket thinkcentre
after i finish malloc I'm going to do smp spinup
that'll be fun
marble salmon
http://www.gii.upv.es/tlsf/files/papers/ecrts04_tlsf.pdf heres the original pape...
the claim that this is O(1) is just wrong
pine rock
how so?
marble salmon
it needs to walk the bucket list to find a non-empty bucket
that's O(number of buckets)
which is O(log n) if the buckets are exponentially in size
also, the experiments only compare against bad algorithms
the established theory and practice is that first fit (or algorithms similar to first fit) performs best in practice
so it'd be very surprising if this "buddy like" strategy outperforms first fit
they only compared against a first fit which walks the free list
which is just stupid
Also, the fragmentation properties are quite bad
this allocator can run OOM despite there being a large enough free chunk available
this can't happen with first fit
marble salmon
this allocator can run OOM despite there being a large enough free chunk availab...
that's because each bucket stores blocks with variables sizes that are within a certain range depending on the bucket
during allocation it only considers the bucket that has blocks that are guaranteed to be large enough for the allocation
but there may actually be large enough blocks in the next smaller bucket as well
marble salmon
depends on what you mean by slab, there is not a single slab allocator design
mimalloc is an actually good algorithm and rather simple to implement
pine rock
honestly unless you actually care about designing your own allocator i'd find a crate to do it for you
cyan nexus
btw, if you want you can see how mine works
pine rock
why a crate?
well its rust
cyan nexus
it's pretty easy to understand
and you can allocate a flexible size
and it can request more pages if it needs + dealloc pages if not needed anymore
pine rock
the ones i've tried are ass
you could wrap mimalloc, snmalloc, or any other C library
theres no shame in not doing everything from scratch
near tartan
something is dying very early during boot
before i get to init the fbcon
i might have to build an earlyfbcon that lives right until the switch to the kernel page table happens
without a backbuffer
i dont have serial on any machine sadly
idle flower
i have two machines with serial, one amd phanon x4? and one dell r730dx
near tartan
damn
idle flower
i just need a disk image or cd image
i hope that the ipmi interface passes through serial
keyboard passthrough didnt quite work last time i checked
i would have more if my father didnt throw out a lot of his machines a decade ago
like 8086 and up, nearly every major generation
up to athlon 64
near tartan
i hope that the ipmi interface passes through serial
it should
i have to add iso building to bootstrap
idle flower
i would have more if my father didnt throw out a lot of his machines a decade ag...
he said "its just junk, its old nothing works with it anymore what do you even want such old stuff for"
idle flower
i have to add iso building to bootstrap
i can attach a disk virtually directly
idle flower
i can try
near tartan
if you run install-minimal it'll only build the necessary stuff of the toolchain
but you could just get away with building limine + menix
idle flower
does it work on win11?
idle flower
To build the distribution you will need the following tools installed on your system:
A POSIX-compatible shell
GNU make
curl
To create a bootable image you will additionally need:
dosfstools (for mkfs.vfat)
e2fsprogs (for mkfs.ext2)
parted (for partitioning the image)
sounds like wsl
hoary cave
you need native linux to run jinx
so there's that
wsl1 won't work
i don't think it emulated linux user namespaces
near tartan
bruh
near tartan
lol
near tartan
wsl1 is the syscall translation layer
that's crazy
idle flower
Could you send me the image directly? Setting up WSL and stuff will take sometime
near tartan
sure
I'll add iso build support and send you one later
or @hoary cave could add iso support 🥺
hoary cave
i am doing linux sooo
probably not helpful
i mean it is still cross-compilation so ig
idle flower
Or make your build system portable, but that will probably take way longer than setting up wsl
near tartan
tell that to mint
idle flower
Nice discord lag, thought I continued my last message
near tartan
lol
idle flower
tell that to mint
Because of limine?
spark surge
????
Jinx needs Linux
it's a hard dependency
WSL2 counts as Linux
if you're on macOS you can get yourself a linux VM and then run Jinx in there
near tartan
windows is a lost cause anyways
i should probably document tho that you can use wsl2
modern hamlet
Jinx needs Linux
but why
what does it use that is linux only and no other unix-like
spark surge
no one uses WSL1 anymore anyways
modern hamlet
no one uses WSL1 anymore anyways
objectively untrue
spark surge
well
spark surge
of course it's objectively untrue, but like
near tartan
problem solved
spark surge
lol
spark surge
what does it use that is linux only and no other unix-like
so like
basically
if you want to do reproducibility and encapsulation in any reasonable way
you need to choose on an OS
modern hamlet
if you use wsl1 you're stupid
i use wsl1 because i need to also use vmware/virtualbox vms and they're slow as shit if wsl2 is usable
and i tried hyper-v as well and it sucks too
also Vmmem is a ram hog even if i don't use wsl at the moment
near tartan
i just dual boot windows
modern hamlet
indeed that is a good solution
i got a 2TB SSD recently, i have no more excuse to not consider installing linux
i just dont know which distro and i dont have much free time
spark surge
CachyOS
eternal wharf
cauchy os
modern hamlet
or maybe arch
spark surge
CachyOS is like #3 on distrowatch, i'd call it mainstream enough
modern hamlet
but im trying to lose weight not gain
near tartan
arch or opensuse is what I'd recommend
hoary cave
hoary cave
thread 'main' panicked at /base_dir/.jinx-cache/cargo-home/registry/src/index.crates.io-1949cf8c6b5b557f/system-deps-7.0.3/src/lib.rs:966:94:
TARGET x86_64-unknown-linux-mlibc is not a builtin target, and it could not be parsed as a valid triplet: Unrecognized environment: mlibc
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace```getting her to use that was a pain i wish upon nobody tho
hoary cave
yeah but it uses cfg-expr to try and parse the target
and it fails because cfg-expr doesn't know about our target lol
this shit's so annoying
hoary cave
that's the pain of not being upstream
also cfg-expr by embark??? https://github.com/EmbarkStudios/cfg-expr
crazy
near tartan
🗿
idle flower
I will probably use rust build.rs as a buildsystem, truly portable and you need a rusttoolchain anyway if you want to compile the code os yourself
pine rock
debian is balls
debain unstable has packages made in the last decade
i still build everything from source anyway though lol
pine rock
i should just use gentoo tbh
near tartan
tbh you should try opensuse
#paidshill
what a good way to manage virtual memory allocations? do i just reuse my phys allocator?
how i feel rn
pine rock
what a good way to manage virtual memory allocations? do i just reuse my phys al...
thats what i did, so theres probably a better way 🫃
but to start with im sure its fine
near tartan
okay i hacked in a super early fb console
lets see what's happening...
?????????
wtf do you mean level out of range
idle flower
me still waiting for the iso
idle flower
nah no stress
idle flower
if necesarry i can download it to my nas, ~7TB free
idle flower
nice windows can extract xz compressed stuff
near tartan
install 7z
idle flower
well it worked out of the box
near tartan
yes
idle flower
like start selected entry after x seconds?
idle flower
could you send a image without autostart?
oh nice right on the day switch
with the last image i had lying around limine ignored the keyboard and just autostarted which prevented me from figuring it out how to pass through the keyboard
near tartan
WAIT NO
NON ONONONONONONOONON
PLEASE DONT TELL ME THIS IS A FUCKING ZEROING ISSUE
💔 brooooo
cyan nexus
PLEASE DONT TELL ME THIS IS A FUCKING ZEROING ISSUE
?
near tartan
i probably don't zero allocated physical memory...
cyan nexus
oof
near tartan
that's why the page tables get corrupted on real hw but not qemu
cyan nexus
why on real hw yes but not qemu?
near tartan
because qemu probably zeroes memory itself
im going to use limines randomize memory function to test this
cyan nexus
lol
distant cypress
I have a flag that I pass to the page allocator (__ALLOC_ZERO) that if set, causes alloc_page to memset it to zero upon allocation.
near tartan
?
cyan nexus
it will randomize the mem as it would be on a real machine
near tartan
i already have
near tartan
because i fixed it
cyan nexus
did you enable it after the fix?
near tartan
yes
near tartan
hm it doesn't even get to the early fb on my framework though...
the only thing i can think of is the serial code blocking infinitely
cyan nexus
gdb time
near tartan
yes it was indeed serial
cyan nexus
lol
near tartan
A LOT
cyan nexus
oof
near tartan
takes a solid minute
cyan nexus
how much ram?
near tartan
both devices have 64gb of ddr5
cyan nexus
i still need to understand how to correctly init APICs and X2APICs
near tartan
this is my brothers pc
grave peak
jesus 31 blobs
near tartan
framework moment
grave peak
this is my brothers pc
try it on every suse owned pc
grave peak
if you dont have numa support multisocket will be slow af tho
near tartan
I don't have smp yet anyways
the new allocator can give out 16-bit memory
i wonder how aids x86 smp actually is
the setup at least
grave peak
its pretty good
grave peak
why self?
near tartan
or like
grave peak
ipi the cpu u want to wake
distant cypress
i wonder how aids x86 smp actually is
honestly if you have atleast a half-decent kernel, booting CPUs via INIT-SIPI-SIPI isn't too bad.
distant cypress
what's the procedure?
- you load the trampoline somewhere in the lower 1MiB of memory.
for each CPU: - INIT level assert
- a very tiny microsecond delay
- INIT level deassert
- another very tiny microsecond delay
- send SIPI(vector=trampoline addr>>12), this is why it has to be in the lower 1MiB
- loop until the CPU says it is online or some number of microseconds have passed
- repeat 6 and 7 once
- if the CPU doesn't say it is online, SMP bringup failed
near tartan
what's that init level assert?
cyan nexus
with limine it's even simpler
near tartan
I don't want to use limine smp
distant cypress
yeah we know
cyan nexus
I don't want to use limine smp
oh k
near tartan
i also want to support other protocols so ic can't rely on that
cyan nexus
how do you tell the other cpus the trampoline location btw
distant cypress
the "correct" thing to do is to refer to whatever the old Intel MP specification says. the sane thing to do is to refer to Linux sources
near tartan
how do you tell the other cpus the trampoline location btw
he said vector= addr >> 12
distant cypress
how do you tell the other cpus the trampoline location btw
vector number in the startup IPI
fiery marlin
the sdms also document how you're supposed to start the APs
near tartan
will look at that
do you happen to know which volume?
i have the amd64 manual in book form
distant cypress
I'm pretty sure there are slight differences between modern AMD and Intel, but the old Intel MP stuff should work on both.
grave peak
damn
near tartan
i have obtained a souvenir
full topaz
i have obtained a souvenir
thats crazy
near tartan
lmfao
that did not work out as intended
💔 why is it so slow
nvm
my old slab impl was fucked
near tartan
fyi this was the worst scenario i could be in
kvm off, debugger on, unoptimized debug build, slab allocator allocates page wise with buddy alloc as a backing allocator
grave peak
Lol
near tartan
okay that's MUCH better
still much more shit than before
sadge
okay now the module loader fails for some reason???
cant find symbols
ok bro
this speed kinda sucks :(
wonder if it's actually because i used a bitmap before
not another allocator issue....
storm bobcat
this speed kinda sucks :(
I thought rust was blazingly fast
near tartan
idk what it is
most likely actually the buddy allocator
that's the only thing that's different from the C version
fiery marlin
do you happen to know which volume?
Not off the top of my head, formulate ots somewhere eslry in volume 3 (something about multi processor init), for AMD I'd have to check.
Oh lol
near tartan
FUCKING RUST
OMFG WHO DOES THAT
KEEP YOURSELF SAFE BRO
@grave peak I'M GETTING SOMEWHERE
i can optimize this a lot still
time to write phd memcpy/set impls
grave peak
Nice
idle flower
time to write phd memcpy/set impls
me waiting to instantly copy them
idle flower
Nice
I will try to use mimalloc in my kernel, hopefully it's faster than alternatives
I'm still sitting at the basics
Jumping between them each time I discover that a previous assumption was wrong/suboptimal
At least the spinlocks look solid
Until I have to debug them at least
storm bobcat
I will try to use mimalloc in my kernel, hopefully it's faster than alternatives
mimalloc is insanely fast, but you have to ask yourself whether or not you want to import 10k lines for it
grave peak
Proxima malloc is 200 lines and takes 100ns per allocation
All it requires is a struct page
idle flower
mimalloc is insanely fast, but you have to ask yourself whether or not you want ...
What could possibly go wrong
storm bobcat
Proxima malloc is 200 lines and takes 100ns per allocation
How do you even measure 100ns
fickle elbow
data structure?
fickle elbow
Proxima malloc is 200 lines and takes 100ns per allocation
in reply to this oops
grave peak
i dont understand the question
near tartan
data structure
grave peak
structured data
fickle elbow
how do you keep track of allocated memory
grave peak
how do you keep track of allocated memory
fickle elbow
https://github.com/proxima-os/hydrogen/blob/be9c17ab4f913198833ee54bc9a62ace003e...
per page linked lists?
grave peak
just a field in struct page
near tartan
lol
near tartan
seems like github scans cargo files
i use talc atm for allocations, but it lacks support for an underlying allocator
grave peak
ah
near tartan
and if i split up the phys allocation to use my own allocator, then i'm going to have some issues with that
there is a wip branch which adds that
near tartan
i may actually be stupid
instead of trying to atomically load a thread pointer i can just get the thread id atomically and use that to get the thread pointer
right?
idle flower
more context needed
near tartan
also i should probably just make my own spinlock primitives
idle flower
mine should be on github
but they are ticket based
as i never plan to have queued threads > hardware threads
else the ticket approach fails spectacular
as you cant preemt the thread
except for independent very short interrupts like msi interrupts
my locks should work ™️
they are untested
well i tested them in my head
grave peak
instead of trying to atomically load a thread pointer i can just get the thread ...
yeah
although thats slow
near tartan
is it?
grave peak
in my old kernel i just loaded lapic id and then indexed by it in the array lol
near tartan
well sure, but wouldn't i have to do locking anyways while i work on the thread data
near tartan
preempt disable as in cli?
grave peak
it can be that if u want
but no, most of the time it just prevents preemption but not interrupts
in nt terms that would be a raiseirql
to dispatch or higher
fiery marlin
instead of trying to atomically load a thread pointer i can just get the thread ...
Just curious why you don't want to go a gs relative load for this? (or equivalent on other others)
near tartan
Just curious why you don't want to go a gs relative load for this? (or equivalen...
hard to do safely in rust
fiery marlin
Ah lol
grave peak
no shame in just disabling preemption
it can be a simple refcounter and an if check in schedule()
and in preempt_enable() just go into schedule() if refcount is now 0
autumn jasper
there is shame
grave peak
rust copium
near tartan
i mean i can store a raw Rc in the percpu struct
which gets assembled/disassembled by the scheduler
near tartan
rust copium <:trl:829316641191886888>
well it's a bit pointless if i just do C-isms everywhere
it's a learning experience after all
i don't write a kernel because i think it'll be usable/successful
grave peak
yeah im just internally accumulating reasons as for why rust shouldnt be used as a systems language
autumn jasper
and in preempt_enable() just go into schedule() if refcount is now 0
this is bad and slow
grave peak
u can have a preempt pending bit if u want to be fancy
near tartan
this is bad and slow
what's good and fast
autumn jasper
my apologies im trying to learn to be more diplomatic
Your provided solution is known to be suboptimal
grave peak
u can have a preempt pending bit if u want to be fancy
did u want this solution?
autumn jasper
yeah
essentially
per-cpu bit that says "the current thread has been preempted but this was deferred due to preempt_disable"
grave peak
yeah basically if your timer irq "tried" to preempt you but couldnt because you're in a preempt_disable it can set this bit basically
which is how most people do it i think
near tartan
ah
autumn jasper
yeah basically if your timer irq "tried" to preempt you but couldnt because you'...
or any other case of preemption
like a higher priority thread being readied (unblocked) by an io completion interrupt
grave peak
yea
near tartan
i think i can use this
then just store the thread pointer directly in the percpu block
near tartan
i told you about that
i forgor
eh it should be fine
how does the Arc even do the conversion
hoary cave
it does inner + sizeof(Inner) for into_raw
and from_raw does ptr - sizeof(Inner)
or, well
inner structure of Arc is
struct {
Rc rc;
T data;
}```so, really, all it's returning is a pointer to your data
near tartan
ah
hoary cave
and from_raw just does container_of
hoary cave
yeah it is, of course
near tartan
i can live with that
hoary cave
into_raw keeps the ref count alive so as long as you have a matching from_raw it's fine
hoary cave
async?? holy
hoary cave
it doesn't make a whole lot of sense to treat futures as full blown kernel tasks
they don't need their own register state etc.
but i'm not sure how it should be done so idk why i even talk 💀
near tartan
idk either
lol
well i think i can allocate a worker thread per irq?
then just have an irqaction defer to a future
or idk
hoary cave
async is rust is pretty funny
do you have any idea how it works? or none at all
i still don't know how it works fully but i do have some idea since i worked with it when writing the hel bindings
hoary cave
well, i'm talking about all the language elements, not about writing async code
Future, Waker etc
well, the Waker interface is pretty funny
near tartan
nah no idea
hoary cave
actually idk if rust async in kernel will be good
i wonder if you can get away without most of the allocations
it's kinda annoying you need at least an allocation per future but idk if any other language does it better
near tartan
actually idk if rust async in kernel will be good
that's why im doing it
trying to see if it's feasible
if it doesn't work out ill just drop it
hoary cave
one thing i know is a bit shitty is that they don't get an AsyncDrop or anything similar
which sucks if you have resource clean up that needs to be async
but it's not really something that only sucks in-kernel, also sucks in userspace
near tartan
rfc when
idle flower
i can send you my design for thread/core local stuff
near tartan
do
hoary cave
DO.
autumn jasper
What do you want to use async rust for in your kernel
autumn jasper
Doubt it
near tartan
async io or something like that
idle flower
pub struct CoreLocalPointer();
impl Deref for CoreLocalPointer {
type Target = CoreLocalData;
fn deref(&self) -> &CoreLocalData {
unsafe { &*cpu::get_cl_pointer() }
}
}
impl DerefMut for CoreLocalPointer {
fn deref_mut(&mut self) -> &mut CoreLocalData {
unsafe { &mut *cpu::get_cl_pointer() }
}
}
pub struct CoreLocalData {
pub(super) arch_data: [u8; 64], //TODO move it into a concrete type definded in arch::*
pub core_id: u64,
pub core_csprng: CSPRNG,
pub system_plm_offset: VirtLv1PageAddress,
pub process_plm_offset: VirtLv1PageAddress, //Move to a dedicated struct later
pub global_data: Arc<GlobalData>,
pub thread_data: TlsData,
//pub process_data
}
impl Drop for CoreLocalData {
fn drop(&mut self) {
panic!("Undroppable Value Dropped")
}
}
pub struct TlsData {
pub process_id: u32,
pub local_heap: ProcessNumaHeapAllocator,
pub held_locks: AtomicUsize,
pub yield_to_scheduler: AtomicBool,
pub current_plm_offset: VirtLv1PageAddress,
}
maybe i should wrap TlsData wit Arc<> tbh
but TlsData isnt supposed to be accessed from outside the owning thread
autumn jasper
managarm uses async in kernel
They don't use rust async in the kernel
eternal wharf
i mean if you grep for co_await you'll find it in the kernel as well
hoary cave
They don't use rust async in the kernel
not rust async, no, but c++ async yes
using rust async wouldn't make a lot of sense given the kernel is not written in rust
hoary cave
glad we agree
autumn jasper
which is what i was actually talking about
near tartan
hm
near tartan
when?
hoary cave
i thought all the allocations you needed was the Box around the future if you want to store it since it's a trait
idle flower
afaik Rust explicitly doesnt do implicit allocations for a lot of things including Futures
near tartan
@hoary cave didnt you say that there's no asyncdrop?
hoary cave
i don't think there is
near tartan
is that an rfc?
hoary cave
behind a feature flag
near tartan
hm
hoary cave
idk if that will work very well but you are free to try
hoary cave
but it seems to be an accepted feature that's being worked on
near tartan
seems like its a bad idea
autumn jasper
i dont even know if i actually want async in the kernel
youre basically building the world so i dont understand the use of something designed to hide the world from you
hoary cave
idk if it's a good choice for kernel code
near tartan
in a kernel context, most things are "async" by definition
like irq handlers
so if that works with normal functions I don't see the need
seems like a pointless abstraction
near tartan
how should i store struct pages for multiple memory regions? do i have an array per memory region or just store one big array (even if it contains holes)?
like, if there's some huge holes in memory between two usable regions, how do i represent that?
hoary cave
like, if there's some huge holes in memory between two usable regions, how do i ...
by not allocating the backing physical memory for this virtually contiguous array
near tartan
oh duh I'm stupid
could've thought of that
so i assume i would also need an allocator for virtual memory, right?
hoary cave
maybe, this "big array" can just be statically placed somewhere
near tartan
hm
hoary cave
i don't think you can allocate memory without a physical allocator first though
near tartan
so at a fixed address like 0xffff_9000_0000_0000?
hoary cave
yeah, well, the problem is also aligning the memory for page tables i guess
near tartan
hm?
what i could do is having a bootstrap page table with the single purpose of building the actual kernel page table
e.g. in the limine-specific init code
hm that's getting quite close to having an eir-ish prekernel
near tartan
the hhdm is pretty much limine specific
hoary cave
well, yes, but do you want to map any memory you access in your kernel manually?
for certain things it's easier to access physical memory through a direct memory map
hoary cave
you need to allocate the page tables needed when mapping the page array
near tartan
but this is a calculation i can do beforehand, right?
hoary cave
maybe, but i don't think it's so simple
you might wanna map some of that memory using 2MiB pages
i guess it is a pretty simple calculation after all
maybe i was overthinking it
near tartan
as i understand it, i just subtract the amount of pages needed to build the pt from a region with free memory?
or what are you thinking of
hoary cave
the amount of pages needed to build the page tables + the amount of memory needed for the page array itself
hoary cave
keep in mind 2MiB pages need to be 2MiB aligned
hoary cave
that slightly complicates the process of reserving memory
near tartan
how so
hoary cave
because it's annoying, assuming you can either "skip" over the non-2MiB aligned memory in the free entry in which case you basically throw away a lot of memory, or do what i did in my shitty kernel, which is ugly
hoary cave
you can, but you're wasting memory
you need to allocate a quite a few page tables you otherwise wouldn't have to
hoary cave
yes, but it doesn't guarantee you'll end up with 2MiB alignment afterwards
what i think is the best way to do this is
let me write some code for that
pseudo code of course
typedef struct { bootstrap_page *next } bootstrap_page_t;
bootstrap_page_t *first_4kib = NULL;
bootstrap_page_t *first_2mib = NULL;
#define LIST_PUSH(head, item) \
do { \
item->next = *(head); \
*(head) = item; \
} while(0)
void do_the_thing(memory_entry_t *entry) {
for(size_t i = 0; i < entry->length;) {
uintptr_t addr = entry->base + i * PAGE_SIZE;
bootstrap_page *page = (bootstrap_page *)addr;
if(is_aligned(addr, PAGE_SIZE_2MIB)) {
LIST_PUSH(&first_2mib, page);
i += PAGE_SIZE_2MIB;
}else{
LIST_PUSH(&first_4kib, page);
i += PAGE_SIZE;
}
}
}
do_the_thing(biggest_entry);
do_the_thing(second_biggest_entry);
you could just have this bootstrap discount buddy allocator before you initialize your actual physical memory allocator
sort them into two buckets, 4KiB and 2MiB
if you run out of 4KiB pages you can take a 2MiB one and split it up into 4KiB pages
or you could do what managarm does, initialize a buddy allocator with all of your physical memory in eir, use it to allocate whatever you need and just pass through the buddy allocator structures to thor
near tartan
a prekernel could work
hoary cave
you don't necessarily need a prekernel for that
near tartan
like an init function then?
hoary cave
good question, i have no clue how to nicely do this, sorry
i guess you could split your physical allocator into two parts
near tartan
lol
hoary cave
the buddy allocator that you could initialize early that handles handing out physical addresses and the one that hands out struct pages
fiery marlin
you can do a bump allocator over the memory map for early physical allocations
hoary cave
but no clue if that's good or not
fiery marlin
since you're unlikely to free any of it
same goes for virtual allocations
you're not going to reclaim memory/address space used by struct page and co
hoary cave
we want to be able to do 2MiB aligned physical allocations