not the region-based one ig? the virtual one we described earlier
#Nyaux
shut laurel
surreal path
not the region-based one ig? the virtual one we described earlier
what
shut laurel
.
surreal path
yea that
shut laurel
i can‘t calculate it in my head rn, but you would need usable RAM spanning from the lowest to the highest physical address, in a single block. That‘s just much wasted memory at best and not possible at worst?
surreal path
i can‘t calculate it in my head rn, but you would need usable RAM spanning from ...
are u talking about the hhdm
or what i was replying to
shut laurel
If you want to not-map any extra memory
surreal path
if ur talking about the hhdm
i already have that in nyaux
and i prefer to keep it
shut laurel
didnt want you to remove it nor map it, a hhdm is good
surreal path
ok so are u talking about the pfndb
shut laurel
ye
shut laurel
right
shut laurel
its not that bad
you‘d have to search through every structure then
surreal path
only takes 2mib
shut laurel
oh wait
surreal path
you‘d have to search through every structure then
no i dont?
shut laurel
true
surreal path
😭
shut laurel
yea yea sorry its too late
surreal path
thats fair lol
shut laurel
yea thats fine, you‘d only need the one freelist linkage pointer. if you have a common base you can just calc the offset for converting to physical address and index by pfn to get the freelist node
surreal path
yea makes sense
surreal path
no
thats wrong
say u have 244141 pages (thats a gb)
- 8
times 8
its gonna be 1953128
bytes
in mb thats
2mb
lol
because the pfndb is only gonna store a pointer
not the actual structure itself
it is basically an array of ptrs in this case
supple robin
whats up
shut laurel
wait im stupid
shut laurel
yea its really too late for my mind
surreal path
yea go to sleep lol
also marvin
its 3am for u
get ur ass to bed
what happened to i wake up everyday at 8am @supple robin 💀
supple robin
supple robin
you wanna bet?
surreal path
mhm
supple robin
nyaux < 1sec boot vs menix userland
cinder plinth
nyaux < 1sec boot vs menix userland
Dufay userland obliterates yall
supple robin
fake
surreal path
tf is dufay
cinder plinth
nah it's true
cinder plinth
tf is dufay
project by Ethan and I
mossy belfry
what
.v
omg this repo is so insanely janky im ngl
sane
oh wow this is a cursed git tree lmao
mossy belfry
💀 this tree is cooked im ngl
just random binaries hanging around
typical stuff
dont you have like a 30gb build directory
what happened
surreal path
its from test v file from vinix
im sure @torn steppe would approve
mossy belfry
too much work for me personally
surreal path
what
mossy belfry
i wonder why the linux kernel doesnt do that
mossy belfry
oh ya
surreal path
your gonna need a build system if u want some actual userland going
mossy belfry
what
i just pray that whoever is building my thing has the right compilers installed and cmake .. && make
mossy belfry
your gonna need a build system if u want some actual userland going
is this for the compiler targets
surreal path
u will need a build system
surreal path
is this for the compiler targets
this is for lit every user program
from bash to ncurses to nano
to the libc itself
they need to be compiled, patched, and packaged correctly and installed into a sysroot
mossy belfry
dang i feel like just making a separate repo for that then
surreal path
this is why build systems like jinx ™️ exist, because your not typing 1000 commands
surreal path
dang i feel like just making a separate repo for that then
yes thats a good idea
check how menix does it
mossy belfry
i suppose i can do that and have scheduled nightly releases of binaries there and just pull those in 💥
surreal path
what
u cannot just use binaries
you need to patch shit
therefore u need to compile from source
mossy belfry
no no i mean i have a separate repo for userland stuff
and then i compile them with patches there
and i use ci/cd to release them as binaries
and then i wget those and build them into the main os repo
mossy belfry
just one massive /usr/bin tarball 
surreal path
thats not how it works
mossy belfry
i could just tar all the userland programs tho
so instead of 100 individual wgets
i just wget the whole thing
surreal path
sure wtv works for u man
mossy belfry
alrighty well this will be fun i love cursed stuff with ci/cd
wasting microsofts time > using a sane solution
surreal path
sure whatever jimbles your jimmys or wtv
now in the meantime
i have some more sleep to catch up on
mossy belfry
aw darn
surreal path
if u can find a repo thats decently okay to port ill port it later after im doing getting the new pmm code working
mossy belfry
if u can find a repo thats decently okay to port ill port it later after im doin...
say mister rayan do you plan to cleanup ur git source tree
and also make other branches in the future
or will it stay this cursed quirky jank
i think this is quite cute
but also mildly inconvenient
mossy belfry
it's cute with a colon three
surreal path
crazy
mossy belfry
variables bro, man, THE_REAL, guy, gotta be my favorites
surreal path
say mister rayan do you plan to cleanup ur git source tree
ok but seriously tho i do not know why it commits clangd kernel/.cache
surreal path
variables `bro`, `man`, `THE_REAL`, `guy`, gotta be my favorites
i genuinely gave up with readablity with nyaux a long time ago
shits like alien language only i can understand
and then i read the code a year later
and i do not understand it anymore
surreal path
real
mossy belfry
i genuinely gave up with readablity with nyaux a long time ago
is THE_REAL something with memory bases
i do not know how to figure out what that is
my lsp gave up too
surreal path
u might wanna do
cd kernel && make clean && bear -- make && cp compile_commands.json ../compile_commands.json && cd ..
mossy belfry
are u unable to remember what THE_REAL is 💀
mossy belfry
cd kernel && make clean && bear -- make && cp compile_commands.json ../compile_c...
im pretty sure u left the compile commands file in the tree lol
ima find it
surreal path
ik
mossy belfry
x tree | grep compile_command.json
├── compile_command.json
``` convenientis there lore to the whimsical variable naming and coding style
like this cannot be easy to write
i would have such a hard time writing programs like this im ngl
surreal path
for me its the easiest
mossy belfry
wowza
surreal path
because i can give no thought into variable names
mossy belfry
talent
surreal path
and just focus on the logic itself
i have trouble coming up with good names for shit
mossy belfry
i have trouble coming up with good names for shit
you have very very silly naming conventions
like sometimes i just giggle while reading the code
it's a joy to decipher
surreal path
if its anyone we should thank, its def qwinci and monkuous
ive put those 2 thorugh my awful awful code
they have suffered
surreal path
3-4
mossy belfry
did it SuckMore before
surreal path
rewrites
surreal path
this is the most advanced kernel atm
surreal path
how bad was it before the current state then lol
awful
mossy belfry
or did the quality remain consisten
mossy belfry
oh deary
were you giving functions unreadble names
memory map function called orange_juice
mossy belfry
oh deary me
surreal path
now i still ignore SOME warnings with nyaux
like the unused shit
but mostly everything is fine with nyaux in terms of warnings
surreal path
frigg
mossy belfry
frigg 👍
maiden ridge
https://github.com/erysdren/quakegeneric
There's one by ozkl the same person behind doomgeneric
mossy belfry
surreal path
anyways in order to make this work im gonna need to bubblesort this garbage
maiden ridge
There's one by ozkl the same person behind doomgeneric
You could try that
surreal path
There's one by ozkl the same person behind doomgeneric
what is it
maiden ridge
quake2generic
surreal path
oh i see
mossy belfry
did you port doom for the meme or is this a game you actually intend to play when you boot the os on real hardware
surreal path
looks reasonable
surreal path
did you port doom for the meme or is this a game you actually intend to play whe...
its three things
- a huge milestone for nyaux
- a meme
- something id actually wanna play
maiden ridge
did you port doom for the meme or is this a game you actually intend to play whe...
It's great for scheduler testing as well
mossy belfry
its three things
1. a huge milestone for nyaux
2. a meme
3. something id actual...
dont you have gpu accel
mossy belfry
It's great for scheduler testing as well
how come
maiden ridge
I gave a copy to a friend and he completed an entire game on it
maiden ridge
Just leave it running and see if it crashes after a while
mossy belfry
real
surreal path
honestly fair
maiden ridge
And probably interact with it
surreal path
also hows polaris
maiden ridge
Keyboard ioctl
maiden ridge
also hows polaris
Nothing much really atm
maiden ridge
Trying to figure out networking
surreal path
holy shit thats based af
mossy belfry
alsmot exactly 420 stars
surreal path
yea networking is fucking pain
maiden ridge
It's dwm
mossy belfry
screenshot said dwm
surreal path
yea fair
maiden ridge
Trying to figure out networking
And also signals
maiden ridge
Nope
mossy belfry
will jinx and allat work just fine on macOS
maiden ridge
Lmao
surreal path
wtf
maiden ridge
how do u have xorg
Nyaux has signals?
surreal path
how tf does xorg even run
surreal path
Nyaux has signals?
no
mossy belfry
my cmake file was crashing out on macOS and i had to fix it for macos
maiden ridge
Oh wait
maiden ridge
You didn't port xorg yet
mossy belfry
typical osdev
surreal path
cooked
wait in polaris do you have like some kind of generic input abstraction interface in the kernel
maiden ridge
None atm
surreal path
cause im trying to think of how input would look in nyaux
maiden ridge
I plan to
surreal path
yes it does
okay so if im gonna need somethin like evdev one day
im just gonna do a funny and
just force ps2 this shit
until is time for evdev
maiden ridge
Since apparently wayland depends on event dev
evdev and drm and of course signals
surreal path
i.e. create some keyboard device file that ps2 might create if ps2 is there
and have some kind of keyboard packet thing
🤷♂️
astral has some keyboard packet thing
i think thats interesting
mossy belfry
how long have u been working across nyaux counting all the rewrites and things
maiden ridge
Yeah
surreal path
how long have u been working across nyaux counting all the rewrites and things
3-4 years
mossy belfry
is this like really far for 3-4 years or something
surreal path
evdev and drm and of course signals
did u impl sockets tho
surreal path
yea makes sense
maiden ridge
Unix sockets, PTYs and Pipes are very similar
surreal path
Unix sockets, PTYs and Pipes are very similar
ptys sound like pain
mossy belfry
dang wow u guys are like senior osdevers nice
maiden ridge
Nope
mossy belfry
o
brisk zenith
ptys sound like pain
implementing ptys is really just making write() calls to one terminal write into the input buffer of another
maiden ridge
^
surreal path
now we are just slightly more unaware ™️
brisk zenith
the rest is trivial if you already have regular terminals
surreal path
implementing ptys is really just making write() calls to one terminal write into...
whats with this slave master shit i keep hearing when i hear pty
maiden ridge
whats with this slave master shit i keep hearing when i hear pty
master is the one controlling and slave is the one that uses the pty
surreal path
master is the one controlling and slave is the one that uses the pty
what does it mean when ones "controlling"
brisk zenith
master is the terminal that the terminal emulator uses
slave is the terminal that the applications use
maiden ridge
The slave reads and writes like it's a normal tty
The master checks the reads and writes made by the slaves
mossy belfry
my os is still in its infancy and i am wasting time writing tons of drivers for specifically the ThinkPad T400 since that is what i want to run it on
i only really spent the past 1-2 months actually working on it and have 2 filesystems and a half (ext2, fat12/16/32, iso9660), 3 disk drivers, scheduler, acpi (duh), and some generic virtual disk and filesystem interfaces
idk i feel like i am moving slowly 💥
surreal path
right
maiden ridge
Yes
maiden ridge
PTYs are simple to implement once you realise it's just 2 pipes
surreal path
i think pipes arent that bad to implement either right
maiden ridge
Nope
mossy belfry
we got ipc before disk drivers
maiden ridge
True /s
surreal path
i cannot be fucked to do anything related to block io rn
maiden ridge
i cannot be fucked to do anything related to block io rn
Polaris needs a fs driver atm lmao
surreal path
i got shit to work on
maiden ridge
There's like NVMe
surreal path
Polaris needs a fs driver atm lmao
fair lmao
mossy belfry
AHCI is super fun trust me bro it is not like every other "Advanced" Intel spec that is just tables that point to pointers to tables to lists to pointers
maiden ridge
Yes
mossy belfry
But no fs driver
wait what lol interesting
aside from ext2, fat and iso9660 what other fs drivers should i make
surreal path
id assume poloris uses tmpfs for its sysroot atm
mossy belfry
i am almost done with my fs stuff
maiden ridge
id assume poloris uses tmpfs for its sysroot atm
Yeah
surreal path
makes sense
maiden ridge
Runs off ramdisk atm
mossy belfry
aside from ext2, fat and iso9660 what other fs drivers should i make
i thinks exFat is kinda worth? idk tho
not very unix of me to write another fat driver
surreal path
i really gotta find a way to compress the initramfs because its always 400 fucking mbs
maiden ridge
ext2 fat and iso9660 seems good enough for now
surreal path
i really gotta find a way to compress the initramfs because its always 400 fucki...
and get nyaux to decompress it
maiden ridge
Polaris ramdisk is ~ 150mb
surreal path
do u compile with -g
mossy belfry
i really gotta find a way to compress the initramfs because its always 400 fucki...
wait so do you literally pack this into the binary and load it at boot
maiden ridge
Yeah
maiden ridge
That's userspace-base
surreal path
wait so do you literally pack this into the binary and load it at boot
no its a module
passed to limine
mossy belfry
ohhh ya
surreal path
which gives it off to the kernel
maiden ridge
Shell, coreutils, the tty console and few programs
Xorg and all included it's like ~ 700mb
surreal path
yea there might be something fucked with my recipes
mossy belfry
which gives it off to the kernel
and then that gets mapped into some part of memory and read and written i suppose
surreal path
i mean limine maps that module for ya
mossy belfry
neat
surreal path
u just do ur shit with it
brisk zenith
i'm looking at the nyaux ramdisk rn and uhh maybe you shouldn't compile gcc with -g
or at least strip it after building
surreal path
honestly fair
mossy belfry
30gb linux kernel compiled with -g and -O0 when
brisk zenith
/usr/libexec/gcc alone is 150M
surreal path
i have it in my jinx config atm where i compile everything with -g
i wonder how i can exclude it in a recipe somehow
mossy belfry
mister rayan is program loading and patching and all that fancy userspace stuff fun
surreal path
not rlly fun when ur depression
mossy belfry
because i have been wasting time doing driver stuff
surreal path
yea
maiden ridge
W
mossy belfry
nyaux doom on real hardware 💥
surreal path
it works on real hw
mossy belfry
😲
surreal path
nyaux doom on real hardware 💥
will do after making the pmm a bunch faster
which requires ps2 at LEAST
brisk zenith
if you don't have any real drivers yet making it work on real hw is fairly easy
because the only "real hardware quirks" you have to deal with are those of the cpu, and those also show up with kvm
mossy belfry
if you don't have any real drivers yet making it work on real hw is fairly easy
but no driver is less fun
surreal path
the only thing u could really call a "driver" in nyaux is probs the fuckin fbdev shit
brisk zenith
the only real difference between a kvm-based vm and bare metal for driverless os compat is that ram isn't zeroed on bare metal
mossy belfry
i'm looking through all the drivers that the singular laptop i wanna get it to run on and i will slowly implement all of them 💥
brisk zenith
and if you have your own bootloader you have to deal with firmware quirks ofc
maiden ridge
Damn
mossy belfry
damn
maiden ridge
Biggest osdev issue
mossy belfry
true
mossy belfry
lavendar face moisturizer is easy fix
maiden ridge
The hardest challenge is finding a partner who loves you-
mossy belfry
no rizz
surreal path
oh god if dennis comes in here and just says that shit im cooked
maiden ridge
Porting chromium nah that's easy
But finding a girl who likes you bro that's hard as fuck
surreal path
real
mossy belfry
The hardest challenge is finding a partner who loves you-
most competent osdev rizz
maiden ridge
LMAO
mossy belfry
this is definitely absolutely not me trust
see the thing is i have had SEVEN straight women ask me out
trouble is
i dont swing that way
it is joever
surreal path
isnt getting a boyfriend easier to be fair
mossy belfry
shockingly not 😔 you would think it would be
it ok tho
no partner = more time for osdev
surreal path
nah for me im fucking dying in the dessert over here, ive had guys ask me out but i dont rlly swing like that
but women
holy shit
maiden ridge
Nyaux hypervisor when
surreal path
i cannot keep one
mossy belfry
dying in the dessert
mossy belfry
i cannot keep one
ts is not a purchasable item vro 🥀
surreal path
Nyaux hypervisor when
when managarm has a good impl of it @thorn bramble get to work ill have that on the cards
maiden ridge
Oh yeah actually
Managarm doesn't have a hypervisor yet?
brisk zenith
nyaux running on bare metal running qemu/kvm running nyaux when
surreal path
iretq is working on it
maiden ridge
How would a hypervisor in a microkernel work even
surreal path
nyaux running on bare metal running qemu/kvm running nyaux when
soon real
surreal path
How would a hypervisor in a microkernel work even
it just does, cause iretq is big brain like that
maiden ridge
How would a hypervisor in a microkernel work even
Type 1??
brisk zenith
How would a hypervisor in a microkernel work even
the same way as in a monolithic kernel?
surreal path
its the actual cpu extension so yea
maiden ridge
the same way as in a monolithic kernel?
I mean fair
but how would you do vmcalls?
Isn't that only restricted to ring 0
surreal path
i do not know tbf
maiden ridge
Isn't that only restricted to ring 0
It would make sense in monolithic since your device is in ring 0
But in a microkernel wouldn't it be a userspace server?
brisk zenith
but how would you do vmcalls?
the vmexit gets passed to userspace to deal with
vmcalls are just a special type of vmexit after all
maiden ridge
Wait hold on how
brisk zenith
the kernel collects info on the vmexit and notifies userspace of it?
maiden ridge
Wait
maiden ridge
Good night
surreal path
gn
brisk zenith
the kernel collects info on the vmexit and notifies userspace of it?
it's the same way cpu exception signals would work in a microkernel
maiden ridge
I mean that I understand
brisk zenith
the kernel collects info on the exception and notifies userspace of it, which uses that info to generate a signal
maiden ridge
But the hypervisor can inform the running guest via an interrupt or something?
Tbf I've not looked into hypervisors alot other than hooking one
brisk zenith
But the hypervisor can inform the running guest via an interrupt or something?
why would it need to notify the guest
maiden ridge
No like see
A hypervisor could have multiple guests running and one of the guests make a vmcall so it has to inform that particular guest running?
brisk zenith
for the most part vcpus can be treated just like any regular old user thread
i'm not sure what you mean by informing the particular guest
maiden ridge
So wait how can the hypervisor pass the info to userspace?
Oh wait it can just inject an interrupt I guess
surreal path
i would like to give everyone in this thread brain damage by making them see this masterpiece
ebon needle
i would like to give everyone in this thread brain damage by making them see thi...
i love tung tung tung tung tung tung tung tung sahur (i already have braindamage)
surreal path
good
ebon needle
good
I have a super duper idea
print sixel image of tung tung tung tung tung tung tung tung sahur in .bashrc !!!
surreal path
no
ebon needle
why 😔
elder shoal
https://www.youtube.com/watch?v=Hsog35L1GMQ
I love italian brainrot
surreal path
I love italian brainrot
real
surreal path
no work today as im too fuckin drained
mossy belfry
bro needs a recharge
surreal path
[ 0.000000][ TRACE ]: him address is 0xffff800000108000, him without virtual 0x108000
[ 0.000000][ TRACE ]: address returned 0x3fc75003
THOSE ARE VALID ADDRESSES
it pages faults with 0x8 as the cr2???
no idea what happened
happening when i return him->ptr?
wait
im stupid
LOL
still page faults
STILL AT AT THAT LINE
i have no idea
[ 0.000000][ TRACE ]: him address is 0xffff800000107fd0, him without virtual 0x107fd0
[ 0.000000][ TRACE ]: address returned 0x7ec0000
[ 0.000000][ TRACE ]: him address is 0xffff800000107fe0, him without virtual 0x107fe0
[ 0.000000][ TRACE ]: address returned 0x7ed0000
[ 0.000000][ TRACE ]: him address is 0xffff800000107ff0, him without virtual 0x107ff0
[ 0.000000][ TRACE ]: address returned 0x7ee0000
[ 0.000000][ TRACE ]: him address is 0xffff800000108000, him without virtual 0x108000
[ 0.000000][ TRACE ]: address returned 0x3fc75003
why
this keeps happening
my code for initing the freelist
surreal path
why am i modifying the memmap length
okay even fixing that
it still dies
fuck
same issue as before
and yes i add the hhdm offset in pmm alloc when returning
as my kernel expects
its the memset
im stupid
it still dies
fuck
okay looking at the log still the return value looks really really wrong
[ 0.000000][ TRACE ]: him address is 0xffff800000108000, him without virtual 0x108000
[ 0.000000][ TRACE ]: address returned without hhdm 0x3fc75003
no nvm its not
the memset isnt messing up the data
so its not the memset
supple robin
bro is him
surreal path
real
ebon needle
what allocator are you targeting
surreal path
what
supple robin
target deez
surreal path
freelist with a pfn database
ebon needle
why not buddy
maiden ridge
freelist with a pfn database
Any resources you're reading? I'm interested in doing something similar
surreal path
Any resources you're reading? I'm interested in doing something similar
@shut laurel and @brisk zenith explained it well enough for me to understand it
i suppose look at eariler messages in this thread
maiden ridge
Hm alright
brisk zenith
it's just a singly linked list
alloc -> remove the front
free -> add at front
you can also add a counter to significantly improve init-time performance
cinder plinth
Any resources you're reading? I'm interested in doing something similar
the concept is well explained in any major OS textbook essentially
the name PFN db comes from NT so you may want to read up on that
but there's also stuff on struct page for linux
surreal path
it's just a singly linked list
since your here do u mind checking out pmm.c, as outlined eariler im experiencing a page fault for some odd reason with returning the address in pmm_alloc https://github.com/rayanmargham/NyauxKC/blob/master/kernel/src/mem/pmm.c
brisk zenith
Sure I'll take a look
surreal path
epic
maiden ridge
the name PFN db comes from NT so you may want to read up on that
Hm alright thank you!
shut laurel
since your here do u mind checking out pmm.c, as outlined eariler im experiencin...
You‘re increasing base but not decreasing length?
surreal path
even changing that it still dies for the same reason all together
brisk zenith
- When allocating the free list, you increase base without decreasing length, causing the free loop to free pages beyond the region.
- In the allocation loop in pmm_init, you unconditionally break out of the loop once you've allocated the pointer array, but this might happen before you've allocated the backing array.
- In the free loop, you set
ptrto(pnode*)entry->base + j- this is equivalent to(pnode*)(entry->base + j * sizeof(pnode)), which I don't think is intended. - Why do you sort the free list?
surreal path
- When allocating the free list, you increase base without decreasing length, ca...
actually i do not know why i sort the freelist??
yes that solved it
thanks for the catch
brisk zenith
Oh and when allocating pagedatabase you only allocate enough room for num_of_pages pointers, but this does not account for holes in the memory map
surreal path
why is it that i always screw up one line
surreal path
Oh and when allocating `pagedatabase` you only allocate enough room for `num_of_...
right
what should i do then
brisk zenith
You need to allocate enough room for align_up(maximum_physical_address, PAGE_SIZE) / PAGE_SIZE pages
surreal path
You need to allocate enough room for `align_up(maximum_physical_address, PAGE_SI...
right so i need to change from using "num_of_pages" to getting the max phys addr and doing that?
brisk zenith
Well no because the current num_of_pages is right for allocating the free list, you just have to add a separate variable
brisk zenith
Yeah
thorn bramble
bro what😭
surreal path
bubblesortfreelist
surreal path
Yeah
uint64_t max_phys_addr = 0;
for (size_t i = 0; i < response->entry_count; i++) {
struct limine_memmap_entry *entry = response->entries[i];
switch(entry->type) {
case LIMINE_MEMMAP_USABLE:
if (entry->base == 0x0) {
continue;
}
for (size_t j = 0; j < entry->length; j += 0x1000) {
num_of_pages++;
max_phys_addr = entry->base + j;
}
default:
break;
}
think this is fine for getting the shit
brisk zenith
Yeah since limine guarantees the memory map's sorted that's fine
surreal path
epic
brisk zenith
```C
uint64_t max_phys_addr = 0;
for (size_t i = 0; i < response->entry_count;...
actually no this doesn't include the last page in the last usable entry
surreal path
oh
thorn bramble
num_of_pages += length/PAGE_SIZE;
max_phys_addr = base + length;
^
surreal path
num_of_pages += length/PAGE_SIZE;
max_phys_addr = base + length;
right okay thats better
brisk zenith
you can change it to max_phys_addr = entry->base + entry->length
thorn bramble
wiht appropriate align up/downs
brisk zenith
wiht appropriate align up/downs
not necessary, limine guarantees alignment for usable regions
surreal path
whihc makes it easy for me real
thorn bramble
true, but if you aren't doing only usable regions then u proabbly need it
also i would not rely on limine doing everything correctly
unless u want to spend the next 30 minutes debugging and filing and bug report instead of working on your kernel
surreal path
okay cool
brisk zenith
limine isn't that buggy lol
surreal path
i trust the mint
thorn bramble
i know
also, you might not want to rely on that assumption for other bootloaders
but nyaux ios probably not getting a mb2 port anyway so
who gives a crap
brisk zenith
oh for other bootloader protocols definitely
surreal path
yes
brisk zenith
for mb2 you can't even rely on the regions not overlapping eachother
cinder plinth
Oh and when allocating `pagedatabase` you only allocate enough room for `num_of_...
cant you sparsely map it?
thorn bramble
oh for other bootloader protocols definitely
and the pmm is supposed to be generic but yeah
brisk zenith
cant you sparsely map it?
yeah but that's more complex than what I'm guessing nyaux is going for
cinder plinth
ah
surreal path
mhm
surreal path
im too stupid to get that working lol
so we doing it the shrimple way ™️
WHY IS TM NO LONGER PURPLE
brisk zenith
it also massively increases complexity because now you can't use your allocator before you've switched to your own page tables so now you need a bootstrap allocator
surreal path
yuh
cinder plinth
it also massively increases complexity because now you can't use your allocator ...
it's fineee
surreal path
i dont think i need the bubble sort
brisk zenith
it's definitely worth it imo
brisk zenith
but not doing it for simplicity's sake is fine
especially on x86, where the holes are tiny enough that you'll probably end up mapping the entire span anyway (except for the sub-4G ECAM region but that's tiny enough to not be an issue)
surreal path
HOLY SHIT
the nyaux speedup
is fucking insane
1000x
HOLY
LIT TOOK LESS THEN 100ms to fucking init the vmm and pmm and arch x86 shit
surreal path
i do that???
surreal path
i do that???
tf
cinder plinth
ah great
surreal path
nyaux isnt THAT simple
brisk zenith
most of that 100ms is probably spent adding every individual page to the freelist
surreal path
makes sense yea
brisk zenith
and you could speed it up quite a bit by adding a counter to the free list entries
so you now only have to add 1 entry for every region
surreal path
id rather not
actually surpingly the vmm takes the most time
not surpingly
wait
we ran out of memory
fuck
thorn bramble
crazy
surreal path
[ 0.000000][ OK ]: vmm(x86_64)(): Above 4Gib Mapped! Mapping Memory Map!
[ 0.000000][ FATAL ]: panic reached :c Function pmm_alloc Reasoning: no memory
i wonder why
surreal path
nvm new bugs!!!
thorn bramble
(new pmm broken???)
surreal path
(new pmm broken???)
ofc it is
thorn bramble
(nooo it can't be)
brisk zenith
Oh btw what's the reason for the extra layer of indirection in pagedatabase, and for allocating the freelist separately? Wouldn't it be simpler to make pagedatabase pnode* with size max_physical_address / PAGE_SIZE * sizeof(pnode), with the free list pointing into this array?
surreal path
wdym
brisk zenith
instead of having pnode freelist[num_pages]; and pnode *pagedatabase[max_page_number], have pnode pagedatabase[max_page_number] and pnode *freelist
that way alloc is just c pnode *node = freelist; freelist = node->next; return (node - pagedatabase) * PAGE_SIZE + hhdm_offset; and free is just c pnode *node = pagedatabase + (ptr - hhdm_offset) / PAGE_SIZE; node->next = freelist; freelist = node;
surreal path
was able to get a stacktrace working
its happening in find_pte_and_allocate2mb
this line
uint64_t *guy =
(uint64_t *)((uint64_t)pmm_alloc() - hhdm_request.response->offset);
yea
im fuckin
cooked
ebon needle
this line
```C
uint64_t *guy =
(uint64_t *)((uint64_t)pmm_alloc() - h...
what is the error code ?
of page fault
surreal path
theres no page fault
i manually put it there
to get a stacktrace
its just running out of memory
surreal path
so it seems like the freelist is broken
Inconsistency at line 11:
Previous -> alloc: 1, free: 32768
Current -> alloc: 2, free: 33021
alloc_diff = 1, free_diff = -253
i made a python scrip to detect an inconsisency in the logs
seems like something is fuckin cooked
we are somehow losing pages???
magically
tender gorge
we are somehow losing pages???
use a electromagnet to scan for them /silly
surreal path
use a electromagnet to scan for them /silly
real
supple robin
sux
surreal path
Inconsistency at line 11:
Previous -> alloc: 1, free: 32768
Current -> allo...
pmm free is not impl'd and will panic on call so this is weird as fuck
surreal path
not only is this is wrong but
we are somehow having some weird ah shit
something happened in between free pages and pmm alloc
the slab allocator is causing this
lmao
the slab allocator after stealing all my pages
its not the slab allocator
its pmm alloc
something is causing pmm alloc to do something cringe
okay so
the line thats causing me to lose all these pages
is in pmm_alloc
head.next = (struct pnode *)him->next;
this one
this looks fucking normal?????
mossy belfry
this looks fucking normal?????
I have a hunch that making your allocator map starting at that address may collide with things that the bootloader maps?
idk I have my heap start much later in virtual address space to avoid that
it's at like 0xffffa.... instead of 0xffff8...
surreal path
oh no thats a physical address with the hhdm added
ebon needle
this looks fucking normal?????
why they doesnt page aligned 😭
maybe there's problem in your initializition ?
surreal path
why they doesnt page aligned 😭
WAIT SERIOUSLY??
huhh
why isnt it page aligned
wait no that makes sense
num_of_pages += entry->length / 4096;
max_phys_addr = entry->base + entry->length;
do u think these 2 lines are wrong and causing this to happen @brisk zenith ?
brisk zenith
no that looks right
surreal path
no that looks right
i feel like the init code is the issue somewhere
as on first allocation we go from xxxxx pages to xxx pages
i.e. we lose a bunch of pages on first allocation
brisk zenith
that shouldn't be possible given how your get_free_pages works
surreal path
that shouldn't be possible given how your get_free_pages works
yea then weird???
brisk zenith
what does your init code look like rn?
brisk zenith
wait really? that link is a commit from 2 weeks ago and the latest commit is from 3h ago with none of the fixes mentioned here implemented
brisk zenith
yeah I was looking at that one but it seems outdated given it still has all the issues we discussed earlier
surreal path
no ur right why is git not recongizing the changes
brisk zenith
anyway what I think is the only thing that could cause this issue is the free list itself being on the free list
surreal path
it doesnt seem so as ive printed every address and they dont overlap
surreal path
https://github.com/rayanmargham/NyauxKC/blob/master/kernel/src/mem/pmm.c
okay @brisk zenith look at master again, i pushed the changes properly
brisk zenith
if (entry->length >= align_up(max_phys_addr, 4096) / 4096) {
for (size_t j = 0; j < sizeof(pnode*) * num_of_pages; j += sizeof(pnode*)) {
pnode **new_ptr = (pnode**)(entry->base + j + hhdm_request.response->offset);
if (j == 0) {
pagedatabase = new_ptr;
}
*new_ptr = NULL;
}
entry->base += align_up(sizeof(pnode*) * num_of_pages, 4096);
entry->length -= align_up(sizeof(pnode*) *num_of_pages, 4096);
done_db_alloc = true;
if (done_db_alloc && done_freelist_allocation) {
goto finish;
}
}```
this seems wrong:
- when checking the length, you assume the pointer size is 1 byte
- you only initialize the first num_of_pages pointers instead of the max_phys_addr/4096 pointers it contains
- the number of bytes you remove from the region is based on num_of_pages instead of max_phys_addr/4096where exactly do i assume the pointer size if 1 byte? also wdym by the second point? sorry my brain is a little cooked rn
brisk zenith
you check if the length in bytes >= the number of pointers in the array
the 2nd point is just that the initialization for loop uses sizeof(pnode*) * num_of_pages instead of sizeof(pnode*) * (max_phys_addr/4096)
surreal path
you check if the length in *bytes* >= the number of *pointers* in the array
RIGHT, so i need to multiply bytes by the size of pnode
brisk zenith
pnode* but yes
surreal path
yea
brisk zenith
also at the done label you should add start->next = NULL to ensure that the end of the freelist is actually marked as such
right now you assume the number of pages you free is the same as the number of pages you allocated for the list which isn't the case due to the pages you removed for the freelist and db
other than that it looks fine
surreal path
```c
if (entry->length >= align_up(max_phys_addr, 4096) / 4096) {
...
wait which line for point number line, its the j for loop right?
brisk zenith
yeah
brisk zenith
I don't think so? it seems to use the right number
surreal path
I don't think so? it seems to use the right number
lets fucking go we gaming
brisk zenith
yoo nice
surreal path
its really fast too
like the speed up is fucking 10000x
this will make nyaux debugging fast as shit
as the kernel can just boot in 1-2 seconds
not incuding the time limine takes to load the initramfs into ram
brisk zenith
the bios disk driver bottleneck is real
brisk zenith
UEFI drivers probably aren't much better
surreal path
yea
kind root
UEFI drivers probably aren't much better
with uefi u can dma directly to high memory avoiding a bounce buffer
so its much faster
brisk zenith
Also technically the int13 lba calls support specifying a 64 bit output buffer linear address
brisk zenith
Yeah hence why I said technically
kind root
i had a laptop which reported support for that 3.0 thing
but it ignored the 64-bit field i think
was a long time ago idr
mossy belfry
as the kernel can just boot in 1-2 seconds
was this not happening prior 😭
surreal path
was this not happening prior 😭
no
mossy belfry
how long was it taking
mossy belfry
nice that it faster now
surreal path
from kmain to userspace is 2-3 seconds
lets fucking go
not including the time it takes to unpack the initramfs + limine loading it
otherwise its like
4-5
ive encountered the funny waitpid page fault again
its more consisent
i get it to encounter every time now
fun
after executing doomgeneric twice
ebon needle
it fucking works
surreal path
what seems to be happening is that the process um is getting unmapped before we can check it
like it could be that the process got unmapped before it could be removed from the list
waitpid tries to check it
and it dies
i am 99% sure this is whats going on
it is probs best to actually use the process lock
surreal path
yo this is fucking crazy
it unmapped it BEFORE it even accesses the lock
so the lock is unmapped
therefore
death.
is there a way to prevent this
?
like even if i spam checks for the process everywhere
it SOMEHOW unmaps it
thorn bramble
is there a way to prevent this
yes
it's called reference counting
also wdym "unmaps"
surreal path
ik
thorn bramble
but yeah, just don't access it lol
why do you keep around a reference to it after it's dead
surreal path
whats happening basically is
reaper.c kfrees it
but syscall_waitpid still holds a reference to it in the while loop
basically
surreal path
because timer interrupt fires and it switches to reaper.c while syscall is still in the while loop
surreal path
also why is there a reaper thread
i do
thorn bramble
any other *nix
thorn bramble
u can literally free it when a parent waitpid's it
or wait's it
i think you can just get rid of it after that
before it's wait(pid)'ed it's basically a zombie
meaning a dead process that still exists
the parent reaps it
surreal path
yea but
thorn bramble
i.e init system
surreal path
im too lazy to change the code
thorn bramble
do it
or deal with stupid bugs
lol
be correct or be wrong and pay the price twice
as a quick and dirty fix u can just disable interrupts around sys_waitpid
but dont rely on that
and go fix the underlying design issue asap
brisk zenith
also why is there a reaper thread
hydrogen has one
thorn bramble
yeah but is hydrogen a *nix
brisk zenith
yeah
brisk zenith
all the regular *nix syscalls are implemented in kernel
which qualifies it as a *nix imo
and you need to have something that'll free it outside waitpid anyway because of SA_NOCLDWAIT
brisk zenith
no that's posix
surreal path
HOW
thorn bramble
i assume linux wasn't posix compliant before 2.6 then 
surreal path
EVEN THO I DISABLE INTERRUPTS
IT STILL COOKS ITSELF??
wait i forgot to save syscall.c
brisk zenith
no that's posix
surreal path
what phone u got
brisk zenith
pixel 6a
surreal path
nice
surreal path
wait i forgot to save syscall.c
STILL
bro
ok genuinely how
atp
ok im gonna play factorio
and then ill gdb this shit after
mossy belfry
from kmain to userspace is 2-3 seconds
is this fastish for a kernel (it gets to the point of setting up devices and modifying disk filesystems)
all in about 1.2B clock cycles
serial writing is the bottleneck here 😡
surreal path
serial writing is the bottleneck here 😡
how so
mossy belfry
idk it just looks like it's slowing it down because I remove that from printf and it is faster
probably because i wait for writes to finish
it's fine idc
I don't really try to make my kernel fast or the code slim I just write things and they turn out to be not terribly slow
turnaround time 💥
surreal path
WTF IS GOING ON
okay on tcg it does this
on kvm it does the fault in syscall_waitpid
UB everywhere
its doing a bunch of random shit
help
mossy belfry
its doing a bunch of random shit
typical sane, regular day in nyaux develompent
surreal path
IT UNMAPPED ITS OWN CODE
AND STACK
I WANNA DIE
the issue has to be elsewhere
not matter what i uncomment in the reaper thread
it still fucking gets worse and worse by the minute
not exitting the thread of reaper thread and just doing a while true do ;;
causes the fault elsewhere
something tells me its exit_thread
nope its not exit thread
mossy belfry
no more stream 😢
surreal path
@brisk zenith this bug is hugely weird, im kinda stuck and i do not understand whats going, a bunch of UB is happening (check thread above). please try and look at the project if you can https://github.com/rayanmargham/NyauxKC
surreal path
okay so the bug pretended to be gone now
this shit is so fucking hard to figure out
surreal path
yea monkuous good luck cause this shit is fucking inconsisent as hell, only thing i can say is look at reaper.c or syscall.c's waitpid or sched.c and cross fingers you find something broken
surreal path
its also inconsisent between tcg and kvm
brisk zenith
oh god a heisenbug
ill look at it later when i can actually test any changes i might make
surreal path
ill look at it later when i can actually test any changes i might make
okay
ebon needle
IT UNMAPPED ITS OWN CODE
lmao
surreal path
ill look at it later when i can actually test any changes i might make
when are you gonna be able to test btw?
brisk zenith
Not sure
Right now I'm dying from heat and I really don't want to turn on the space heater that is my PC
surreal path
yea its quite hot pretty much everywhere
its hot in ireland too
climate change be like
maiden ridge
Damn what all packages are you including
surreal path
Damn what all packages are you including
deps="base-files kernel bash ncurses readline doomgeneric nano ttytest binutils coreutils gcc fastfetch"
maiden ridge
But weird
Since I include a few more packages such as zstd lua and gcon
And the ramdisk for that is ~ 150mb
surreal path
maiden ridge
deps="base-files kernel bash ncurses readline doomgeneric nano ttytest binutils ...
base-files kernel drivers init bash coreutils lua nano ncurses readline tzdata xz zlib zstd gcon
surreal path
`base-files kernel drivers init bash coreutils lua nano ncurses readline tzdata ...
is that all ur shit?
maiden ridge
That's userspace base yes
surreal path
right