Zag

[past dome]

probably

[fallen bobcat]

What'll you do then

Maybe this is not needed if you don't do much memory reclamation in work items and the reaper stuff can be manually called I guess

Cuz the idea is to run work items to try and free memory

[past dome]

high priority work items never do a blocking allocation, they can only free, if it allocates then it needs to be a lower priority one

so the high priority worker threads will not all block on oom

at any point

[fallen bobcat]

The issue is that if a worker thread blocks you want another thread to pick up work while that other one is blocked, so you try spawning another one but that blocks so you're fucked

[past dome]

i understand the issue

[fallen bobcat]

Maybe it's just how Linux does it, Linux spawns other threads directly in worker threads when they need to

But that's unrelated cuz this is still an issue no matter where you spawn the thread

[past dome]

what im saying is that the existing pool of high priority worker threads will always suffice to drain all the work items that free memory

they can block on page frame allocation if they call paged code or touch paged pool but if such pages have been fully removed to disk then evidently there is pageout going on and so pageout (which is not done by these worker threads and never blocks on allocation) will free up more pages and theyll get going again

[fallen bobcat]

And if A blocks and B is woken up as a result and it blocks on creating a spare thread of its own it's not so bad since A should unblock promptly since high priority work shouldn't block for long?

[past dome]

i am too sick to properly interpret what u just said

[fallen bobcat]

Basically what Linux does is that worker threads themselves ensure that there are enough threads in the pool in case they block

But the problem is if that blocks then high priority items can't run until the allocation unblocks

[heavy sandal]

this might be what the maximum thing for the NT queue is for btw, letting it make a bunch of suspended threads in advance to use later. im not 100% sure tho

[fallen bobcat]

No

NT maximum count is basically "how many threads should process items concurrently"

[heavy sandal]

ah

[fallen bobcat]

It doesn't spawn any threads as it's a kernel thing

[heavy sandal]

ah right

the layering is weird

i always forget the order

[fallen bobcat]

And it does that by knowing the threads that are blocked on the queue and letting only that many in

[heavy sandal]

yeah

[fallen bobcat]

And if one thread blocks while processing an item it unblocks another

Linux workqueue is surprisingly similar

It's not as generic but it's practically the same thing

[heavy sandal]

turns out its a good design concept

ig

[fallen bobcat]

ugh it's annoying how I have to half-ass subsystems because i need other stuff then I come back to them

like i cant really do the fancy dynamic worker pool yet because I don't have a thread reaper, but the thread reaper is a workqueue item

so for now I'll just have two worker threads per pool per CPU I guess

[fallen bobcat]

i am coming back to this

this was indeed a good idea

[vital summit]

real

[fallen bobcat]

I have implemented asynchronous TLB shootdowns

This is the only kernel that I know of that does this

worst case scenario it goes synchronous

so it's faster in most cases but in the slow path it's only as slow as regular TLB shootdowns

[fallen bobcat]

wait no I think managarm does this already

ok mine is the only one that is IPI-less

I do have a nasty layer violation though, the depths of the kernel have to call into the memory manager

[long pendant]

it might be solveable

vm maps have a component which logically belongs in the kernel division anyway (at bare minimum, the page table address/whatever you need to program the MMU on context switch)

so depending on the exact design you are doing, put whatever you need for async tlb shootdowns into that data structure, which is of the kernel, and the vm map embeds that as its first member

(this extraction of a ke-side component that's the basis of a vm-side vm_map is one of the changes i have in mind for nukeyronex)

[fallen bobcat]

Well, the kernel just needs to call into mm.process_shootdowns function on a quiescent state

I guess I could make this some kind of kernel facility, but I accept this upcall for now

[long pendant]

what does that involve, does it need allocation or any awareness of the shape of a vm map?

[long pendant]

but that's fair enough in any case

[fallen bobcat]

Not necessarily, it just needs to go through its pending shootdown states and flush them

Right now what I do is that I free the page when it's the last cpu to acknowledge it too

But that could be moved to a worker thread

I guess I could make it a DPC

[vital summit]

how does this work?

[fallen bobcat]

It doesn't send an ipi

[vital summit]

yeah but how do you notify other cpus

is this some arm meme

[fallen bobcat]

By not sending an ipi

[vital summit]

😭 can you stop vagueposting

im genuinely interested

[long pendant]

is this for userspace memory as well as for kernel-space?

[fallen bobcat]

Both but right now only kernel

[long pendant]

(kernel-space easier in principle since you shouldn't access it once it's been officially freed so you can lazily shootdown the window that's no longer mapped and once it's been shot down by everyone, you can reuse that span of virtual address space)

[long pendant]

that makes it very interesting since it's more challenging for userspace

i suppose that's why the page free is deferred

[fallen bobcat]

Yeah

[fallen bobcat]

Basically it does all of this without sending an ipi

Nah but fr wait a sec

[vital summit]

[fallen bobcat]

@long pendant if you're curious too

[fallen bobcat]

Also idk if arm has that (probably) but x86 has invlpgb

Which is based af

[vital summit]

yeah arm has that

but wait invlpgb isn't available even on quite recent hw

like, it wouldn't run on my server

[fallen bobcat]

Yeah I know

I don't use it

I just know it exists and it's cool

Maybe I could support it eventually

[vital summit]

oh

[vital summit]

I don't get this

that would mean that the address stays alive in the remote cpu until the worker does its thing

unless I'm being dum

let's say you're doing an munmap, how do you handle that?

does the caller wait for all cpus to ack the tlb shootdown? that wouldn't be very async

[fallen bobcat]

No

[fallen bobcat]

Yeah so

[vital summit]

man

idk how to explain

i don't get what the point of asyncness is

[fallen bobcat]

You don't block on CPUs

[vital summit]

when do you not want a sync shootdown

[fallen bobcat]

Imagine you're running on a system with like 64+ cpus

You don't wanna send an ipi to each one of them and wait synchronously for the shootdown to complete

You just notify them and they'll do it eventually

There was a paper that implemented it in Linux and there were performance improvements

[vital summit]

i get that, but isn't that not what you want

[fallen bobcat]

heck even 16 CPUs

[vital summit]

like, what's the point of doing a shootdown but not serving it immediately?

[fallen bobcat]

so you dont wait for it??

when you free a kernel heap address for example

[vital summit]

ohhhhh

so for userspace you're doing an ipi, right?

[fallen bobcat]

no, that could work for userspace too

[heavy sandal]

it relies on noone using that virtual address until the async thing is done, but usually you arent immediately remapping the same vaddr to something new anyway

[fallen bobcat]

unless there's a weird guarantee that mmap reallocates the last freed address

[vital summit]

but isn't there something in munmap that forces you to invalidate mappings on all threads before it completes?

[fallen bobcat]

wait fuck does this break POSIX

yeah

i guess that could be done synchronously

or you just dont care

[heavy sandal]

rip

[vital summit]

just do the shootdown exclusively when munmapping, everything else can be done async afaict

[fallen bobcat]

yeah

I'd have to read the paper to see if they talk about that

but in practice not much software should rely on that

[heavy sandal]

even if theres cases where this setup doesnt work, its still helping in most cases

also its neat

which is worth something in a hobby space imo

[vital summit]

Further references to these pages shall result in the generation of a SIGSEGV signal to the process.

that's all it says

[heavy sandal]

you can partially async still then

because you only need to sync on cores that are already running that process

because if a core isnt already running that process it cant start doing so without doing a context switch which would then trigger the lazy local flush

[fallen bobcat]

idk the paper doesnt seem to mention it

[vital summit]

maybe you don't have to care really?

[fallen bobcat]

yeah

that's broken code anyway

[heavy sandal]

love to just say fuck you to the spec lmao

[fallen bobcat]

also this is from their STUPID implementation and my GENIUS one should be faster

[heavy sandal]

lmaoooo

[vital summit]

i mean you can just await the shootdowns on the calling cpu

that would only be blocking one cpu

[fallen bobcat]

stanford nerds

[fallen bobcat]

yeah, that's doing it synchronously

[fallen bobcat]

this is partly true btw

[vital summit]

is it if you're not sending an interrupt?

[fallen bobcat]

they missed obvious stuff which could make it faster

[fallen bobcat]

but then that would be tricky

[heavy sandal]

semantics tbh. id say the key difference between normal tlb and this isnt the no ipi its the no blocking

[fallen bobcat]

yeah

I could send an ipi as well

[heavy sandal]

because the act of sending an ipi you end up waiting for the ipi to be delivered even if you dont wait for the handler to finish

[fallen bobcat]

managarm sends an ipi but still does it asynchronously

I do it on quiescent states instead

I could do some fancy expediting if I wanted to by sending an IPI if a CPU is blocked or something

[heavy sandal]

tbf the main benefit of the sigsegv isnt that normal code detects it its that it helps you find broken code to fix in development

[fallen bobcat]

yeah ok I asked a clanker to be sure I didnt miss anything

[fallen bobcat]

the only problem would be in a multi-threaded application that does a use-after-munmap from another thread from which it was munmaped

cuz the local CPU is always flushed instantly

[vital summit]

figured as much

[heavy sandal]

yeah, but that is a bug that can be useful to have get caught when trying to develop programs

[vital summit]

but honestly the timing would have to be quite tight for this to go wrong

[fallen bobcat]

there's a short time window where it wont get caught

but it's short

[heavy sandal]

yeah

[fallen bobcat]

like, < 10ms

[fallen bobcat]

this would be useful if memory is lacking

then you just send a reschedule IPI to all CPUs with next_thread set to NULL to kick the CPUs to do their shootdowns

mhm, I have to think of how i could make this a kernel feature instead of mm

if I make the allocator locks blocking, I'll have to do freeing in a worker thread. I can expedite the freeing in DPC context if trylock succeeds however

[fallen bobcat]

btw nukeyronex when?

you need to become a NEET like me so you can work all day on keyronex

[languid canyon]

Whats a neet

[fallen bobcat]

https://en.wikipedia.org/wiki/NEET

[languid canyon]

Oh

What do u do?

[fallen bobcat]

well I'm attending uni in fall

but rn it's summer and there are literally no jobs 😭

I've done like 4 interviews already havent been called back once

and it's shitty part-time jobs

and I wanted to do gsoc this year but since this year they removed eligibility for quebec

I've tried freelancing on fiverr but I've only gotten bots

[languid canyon]

Lol why

[fallen bobcat]

idk new law and google's payment processor didnt care to follow it

so they just removed eligibility

[languid canyon]

Bruh

[heavy sandal]

oh thats what that means? ive heard the term so much but never knew what i meant lol

[fallen bobcat]

and it's only since this year

fricking annoying

quebec has a bunch of laws that organizations dont care to follow, so you often see e.g in coding contests that participants from iran, north korea, etc. and quebec are banned lol

[long pendant]

i have some very messy work locally, including some experiments with RISC-V hypervisor support

but i've been temporarily distracted by working on an algol 68 compiler

[fallen bobcat]

hey @rocky yoke I noticed in managarm code that you allocate ShootNodes when doing a shootdown, how is that not an issue if you run out of memory?

because those nodes are needed to reclaim the memory

https://github.com/managarm/managarm/blob/master/kernel/thor/generic/thor-internal/arch-generic/asid.hpp#L263

[vital summit]

i don't think managarm particularly cares about that

on OOM it also just panics

*yet

[fallen bobcat]

i guess one way to solve this would be to keep a few statically allocated shootnodes around

[rocky yoke]

yeah, i think the long term fix is to just block if the allocation fails and do a synchronous shootdown

aside from that, shootdown nodes do use a different allocator than the general heap

they're allocated directly from the higher half direct map

OOM on that allocation is still possible but only if there is truly not a single page left

[fallen bobcat]

I'm mostly worried about slot reuse latency on my part

If it takes too long to reclaim memory/the invalidation slot then it'll fallback on synchronous more often

There are maximum 64 outstanding shootdowns per cpu

this is running fireworks (I put large data in the fireworks data on purpose so that it gets shot down)

so roughly 10:1 async:sync ratio

which is not that bad

but that's not representative of a full system running multiple programs and doing lots of munmaps

[fallen bobcat]

oh wow just marking a softint pending on the remote CPU speeds it up quite a bit

[fallen bobcat]

I think how it will work is that the async shootdown machinery will be in ke, and when a shootdown completes its state will be put into a kernel queue, which gets picked up by the memory manager in a worker thread, which actually reclaims the memory

shit I left ping turned on

[fallen bobcat]

though this makes a question arise: what priority should this worker thread be? It has to be somewhat high so that other async shootdowns can proceed (as slots should be promptly reused), but not too high as to prevent other threads from running. I could make it high priority but still have some kind of timeout on the memory I reclaim as to avoid using 100% of the CPU

also obviously it shouldn't hold onto memory for too long

I guess it's not really an issue considering async shootdowns are an optimization, and if even 2/3 of shootdowns are async then it's good

[fallen bobcat]

I guess if memory pressure is high when the memory manager is called to relieve the pressure, it does it itself

could also have more threads

[fallen bobcat]

holy shit I found a very fast reclamation glitch instead

that is very cool

the unmap code returns a list of backing physical pages threaded through struct pages

you can just take the tail of that and put it at the beginning of the page freelist in a single operation

[past dome]

how come youre conflating unmap with freeing

what if its a shared page

or what if its a private page but it needs to be swapped out

[fallen bobcat]

It's not generic unmap, it's a different function that'd be called here

If no physical page needs freeing because it's shared or something then it'll just not free it

This is moreso delayed reclamation of memory vs just shootdowns

[fallen bobcat]

What's the problem with this? Is that the page must be instantly freed? In that case you'd just fall back on the synchronous path

or the vma shouldn't be reclaimed I guess

[fallen bobcat]

crazy how i wrote the code in a day but I havent had the motivation to clean it up for like 4

doing it rn

[fallen bobcat]

finally pushed

💀

next steps are cleaning up the overall codebase, make pmm shittier then do more process stuff

[cyan prism]

why are you making the pmm shittier?

[vital summit]

so that reverting is an improvement

[fallen bobcat]

It's too complex for what I wanna do with it, I'm moving to a simple freelist

[cyan prism]

Ah I see, fair enough

[fallen bobcat]

casually statically allocating 1 mib

[fallen bobcat]

💀

[languid canyon]

What is this

[fallen bobcat]

dragonflybsd lwkt tokens

I think I can implement them better with turnstiles

cuz this is nasty

maybe I'll discuss this with dillon on irc

[languid canyon]

What is that

[fallen bobcat]

basically locks that drop when you block

and reacquire automatically when you wake back up

[languid canyon]

Wait why is there no indentation after if

[languid canyon]

Cool

[fallen bobcat]

nvm

[fallen bobcat]

mhm ok I will implement serializing tokens eventually I think

they're cool

but I'll do more urgent stuff first, cba to design them rn

[fallen bobcat]

this is tricky and I'm unsure if this is the right way to go

[past dome]

These always struck me as an implementational tool for reducing the invasiveness of changes required to improve locking in old code

Rather than something good on their own

I don't think they're necessary in a new kernel at all

[fallen bobcat]

yea but isn't it often useful to release locks before blocking, and that might happen somewhere deep in the call stack so you have to pass locks around? or is that just a code quality problem

[heavy sandal]

ime usually only with condition variables

where you hold a mutex and then do an atomic signal-and-wait on the mutex and condition variable

blocking on other stuff while holding a lock is usually a code smell imo

[fallen bobcat]

I agree that they were mostly implemented to clean up the code rather than anything else, dillon cites freebsd-5 has having that issue deep in io code

[past dome]

the way i write kernel code tends to result in a deeper part that decides blocking is needed returning to an upper level who sees that and then releases the locks he grabbed and does the blocking himself, for example page waits or IO completion waits

so the lock is usually released in the same function that acquired it

[fallen bobcat]

that makes sense

afaik tokens have always been a thing to write cleaner code anyway

[fallen bobcat]

yea which is why I said you need to release the locks

[heavy sandal]

i meant more like any code where youd need something like this is smelly - its easy to just manually release, then block, then reacquire

and doing it manually is more readable and verifiable

(which is part of why i prefer having an explicit atomic signalandwait operation available over condition variables etc that magically unlock on certain conditions)

[fallen bobcat]

I don't think condvars are really useful

[heavy sandal]

shrug

more using them as an example of a similar release-and-block operation

[fallen bobcat]

what do you use them for

[heavy sandal]

i dont have them yet in my os, though ive used them extensively in c#. good for blocking queues if you suck at atomics

[fallen bobcat]

yeah in userspace they're more useful

also I'm coming up with a zig trick to assert interfaces

isntead of doing ugly if !(@hasDecl(..))

[heavy sandal]

if youre just issuing an error at the end of that i usually dont bother tbh, just let the compiler error for decl not found do it

[fallen bobcat]

yea but it's cool because I can have a "schema struct" which helps guide implementers

[fallen bobcat]

kool

[fallen bobcat]

finally added all the thread machinery for the console flushing

that'll be a shitty kernel console for showing logs only, the proper vt100 terminal will be in userspace

[fallen bobcat]

there's so much work to do 😭

i zoomed too far on emacs and now it almost crashed my system bruh

[languid canyon]

vro has the skyrim kernel

[fallen bobcat]

ur just made because your font is not as cool

[fallen bobcat]

also why does ultra have almost as many lines as my kernel wtf 😭

no it has more actually

[languid canyon]

you're looking at the current upstream which doesnt have that many

[fallen bobcat]

i blame it on header files

unironically

[languid canyon]

i have a lot of test lines

[fallen bobcat]

I count in kernel/

only

[languid canyon]

the downstream code i have is 17k

but it also has more tests

and still does literally nothing

[fallen bobcat]

not really a fair comparison since not the same language

[languid canyon]

how many do u have

[languid canyon]

yeah maybe its the headers

[fallen bobcat]

my code is more complex tho

L

[languid canyon]

is that good or bad

[fallen bobcat]

dunno lol

[languid canyon]

lol

id imagine less complex good

because easier to read or smth

[fallen bobcat]

the temptation to work on a useless kernel debugger...

[languid canyon]

Lol

Do it

[fallen bobcat]

most useful thing ever

yeah no ok I dont really see any use for this that I cant do with just prints, maybe in the future I'll come back but for now I won't stage this lol

[languid canyon]

finally, kernel shell

[fallen bobcat]

I think it'd be genuinely useful but I cba to implement everything atm

I'd rather work on interesting stuff

[languid canyon]

i guess it could be in some ways

but at the same time gdb can probably do most of that itself

[fallen bobcat]

yea but gdb doesn't like zig a lot

[languid canyon]

ah

[fallen bobcat]

implement the debugger for me

i will pay u

[languid canyon]

what makes you think im qualified

[fallen bobcat]

uhh i looked at your CV

good enough

obviously not as good as me tho

[languid canyon]

damn

yes

[fallen bobcat]

nah ok I really need to unshittify the whole thing

next goal is making a thread reaper

I saw NT do some lockless singly linked list, I will do that too

[past dome]

check how mintia2 do it

[fallen bobcat]

ive stolen so much from mintia already I can't 😭

nah jk I havent stolen that much

[languid canyon]

just say you're continuing the monkuous port of mintia to C

[fallen bobcat]

but I plan on stealing continuations though, in exchange you can get the kool async tlb thing

[fallen bobcat]

https://github.com/xrarch/mintia2/blob/main/OS/Executive/Ke/KeCustodyList.jkl ah so this is just a generalization over the NT thing

[past dome]

its a pattern i thought would come up a lot

wrong reply

wtf

anway

it did

[fallen bobcat]

object reaper and thread reaper

idk what else

this does sound like a good primitive to have though

[past dome]

i also use them for giving threads that donated their stack bc they blocked on a continuation a stack when theyre about to be switched to and the per-node stack cache is empty

[fallen bobcat]

it has no kernel magic so I'll just make it an rtl thing i think

I'll call it handoff list

"OK you can copy my homework but don't make it too obvious"

[cyan prism]

I have a local debugger (yes, its a kernel shell) and found it pretty handy on some hardware. Also a nice to have if you're testing on a system you cant get serial in and out of.

[fallen bobcat]

yeah I do think it'd be useful, but I'd rather clean things up and improve things first. Also eventually I will have a prekernel of some sort that would only load the debugger stuff conditionally I think, so maybe I'll wait until I have that to work on it proper

[solid marsh]

stop waiting and start working

[fallen bobcat]

I am working

but on other stuff

my TODO list is huge

i should probably write it down, it's all in my head atm

[solid marsh]

do you have a gdt?

[fallen bobcat]

yeah lol

[fallen bobcat]

best gdt

[languid canyon]

smhmh no PER_CPU

[solid marsh]

then your kernel is done then

[fallen bobcat]

you have one gdt per cpu?

[languid canyon]

ofc

[fallen bobcat]

oh yeah for the tss

[solid marsh]

it's what you have to do

[fallen bobcat]

but i dont have a tss

[languid canyon]

well u can have a spinlock

so u serialize TSS loading

[solid marsh]

that sounds

terrible

[fallen bobcat]

I really dont like working with architecture stuff

it's so boring

[solid marsh]

tbf for me it's the other way round lmao

[languid canyon]

^

[fallen bobcat]

how

[languid canyon]

i kinda hate the general kernel stuff

like allocators and shit

so boring

[solid marsh]

same

i get it, it has its interesting problems

[heavy sandal]

same

[fallen bobcat]

you guys should join the project and only work on improving the architecture stuff

[solid marsh]

rn im working on having huge pages in my os and im yawning every 3 seconds lmao

i want to do whatever , just not memory management

[fallen bobcat]

I'm not bothering with huge pages

can of worms

[solid marsh]

im just going to have a list of pages available from boot and that's it

[heavy sandal]

I don't even remember if my os has large pages yet lmao

[fallen bobcat]

I support them architecturally but they're for direct mappings

mostly

if you map a contiguous range of virtual and physical memory it'll use the larger pages it can

[solid marsh]

yeah that's good

(what you should do)

[fallen bobcat]

and this is all abstracted away and super easy to port to other archs

you just define a few constants

*if you're using an architecture with page tables as trees

[solid marsh]

i overlooked the fact that all modern arches have something like a radix tree for paging and right now my memory manager is super gangrenous

[fallen bobcat]

for UM I do this

which is so bad lol

well it's not that bad

but surely not the best way to do it

I'm thinking of getting rid of it, it has bitrotten a bit and idk if it's really that worth it

[vital summit]

[solid marsh]

everyone knows that the only right way to make a gdt is to make a define for each bit and make a gigantic macro soup

[fallen bobcat]

since when do you do C

[vital summit]

why not

[fallen bobcat]

zinnia in C???

[vital summit]

no

[fallen bobcat]

when

[vital summit]

it's a small microkernel i just use to mess around with

it boots but that's it

[fallen bobcat]

oooh

[vital summit]

it's not very sophisticated

dw about it

[solid marsh]

isn't it easier to literally

scan the memory for the rsdp

[vital summit]

and do that how?

[solid marsh]

in bios systems the rsdp can be found in the ebda or in the bios rom

so what you have to do is to find RSD PTR in these regions

[vital summit]

1. my stuff is always uefi only
2. you can't just read arbitrary physical memory

mmus exist

[solid marsh]

well

[vital summit]

ohhh i see the confusion

this is in userspace

[drifting blade]

What do you use for debugging?

I've gotten into Zig and the debuggers aren't that great, both gdb and lldb work poorly

[fallen bobcat]

I write good code

[vital summit]

lol

[drifting blade]

Until I find a good solution I def wont osdev in Zig it would be a pain

[round tangle]

Has it turned out to be more effective to have several per-CPU work queues rather than a global one?

[solid marsh]

depends on the purpose really

[fallen bobcat]

Honestly debuggers are rarely useful

[drifting blade]

I disagree, print debugging is so annoying and sometimes I can't print debug

[fallen bobcat]

Have you tried staring at the code

[drifting blade]

Staring at the code can work but for complexer things a debugger is nice to step through the program

[fallen bobcat]

i did something legally dubious but very cool

running local elixir cross-referencer on WRK

[fallen bobcat]

I just saw this and idk yet, Linux does do so

[fallen bobcat]

thread reaper stuff is done

i can now run without ooming

stableish memory usage

low synchronous shootdown count

pretty good

[fallen bobcat]

im making the allocator blocking and some stuff is tricky

especially because free needs alloc

i just need to take in a NoWait flag

[fallen bobcat]

yeah the problem is that there is memory in the heap but no physical memory

i think

[fallen bobcat]

huh another tricky problem is that for mapping heap memory, you need to take into account that mapping pages can create PTEs and fail to do so

well, that's just for mapping stuff in general, but the tricky thing here is that you dont want to take a lock across potential blocks

this is a place where something like serializing tokens could be useful, but I think it can be designed around

[fallen bobcat]

yeah ok i really need to get into MM design cuz this sucks

[rocky yoke]

I've encountered the same issues when adding OOM safety to Managarm (which is not done yet)

I think in Managarm's case I can probably move almost all allocations out of locks

but that's most likely more difficult for a monolithic kernel

[languid canyon]

why?

[cyan prism]

probably for metadata if the free is in the middle of an in-use area?

[languid canyon]

oh is this talking about the vm area manager thing

[cyan prism]

I believe they're talking about the heap

hmm no, that sounds insane.

I'm not sure actually

[fallen bobcat]

it allocates a magazine if there's none

If that fails then it just frees on the slab layer

[fallen bobcat]

I think the way to solve it is to manipulate ptes manually, you go through each pte of your virtual address and try to allocate it if it doesn't exist. Then you go through each pte and allocate a physical page for mapping the memory at. At each step if a physical memory allocation fails you have to drop the locks then wait for free pages manually

[rocky yoke]

what you can also do is make sure that your cpu-local magazine has enough objects (e.g., pages) for the worst case before taking locks

[languid canyon]

do u explicitly map every allocation somewhere instead of pointing it to hhdm?

[fallen bobcat]

For large allocations yeah

[languid canyon]

does your allocator auto transform into vmalloc for large allocations?

[fallen bobcat]

Yeah, I don't do cpu local page lists though but that also works

you'd need to hold the page list lock across the operation though

[fallen bobcat]

yeah

[languid canyon]

I see

yeah i guess that may be problematic

[rocky yoke]

you have the same issue if you have two allocators

[languid canyon]

nah i mean i can see where the issue arises for him

it didnt make sense if it just pointed to hhdm

[fallen bobcat]

There's the same problem for any userspace mapping too

[languid canyon]

do u preallocate the entire mapping at map time?

[fallen bobcat]

yeah I'll have to do that

unless you mean userspace

[languid canyon]

yes

[fallen bobcat]

Userspace will demand page

But the issue could still arise if you have like 2 pages left

Or if you somehow pre-page a contiguous range of memory because the access pattern shows it is beneficial to do so, then it'd be more likely

[fallen bobcat]

what gets tricky here is the locking, you want to try to create a higher-level PTE so you have to drop map lock before allocating, but then another thread can come along and also try creating the same PTE, which will race. One solution to this is have some kind of lock bit in the PTE that you spin on to protect against modifications on that particular PTE but I'm not sure how good that is. You could also have one alloc lock and one free lock, where you take both first, then when you have to allocate memory you drop the free lock but keep the alloc lock held. The alloc lock protects the creation of PTEs and I guess free lock protects VA creation and deletion, not sure

@long pendant i might require your expertise here

intuitively the PTE lock thing sounds good, but the locking discipline might get weird

especially now every time you have to traverse the page tables you have to go lock each one, I don't think there'd be any kind of lock ordering issue here but I might be missing something

(btw when I mean PTE I mean any page table entry, not just the leaf one)

I guess one thing you can do is bail out if you wake back up from allocation and you notice the PTE was already filled, then you just free the page you just allocated

[long pendant]

I explored the problem space once and didn't get a satisfactory solution

[fallen bobcat]

I think this is sensible but maybe not the cleanest way to do it

[long pendant]

My plan was to put the lock bit in the vm_page_t (thus one per page table) but then I also had cause to introduce a "spin pte" created, I think, under that lock, but spun on till it turned into a different kind of PTE - i dont remember the exact details but maybe I'll remember more when Im home

[long pendant]

Yes I think that's exactly one of the areas I considered the spin pte for, but concluded it was very stupid for that - just tolerate having to recheck your work and do things like allocation tentatively

I ended up not pursuing a per page tablle lock model but separated a creation and stealing lock which does enable me to allocate without having to recheck

Page allocation might steal someone else page under her stealing lock - while creating a new valid PTE is usually done under creation lock instead

Creation lock is a poor term which doesnt capture the subtlety. But ill elaborate on this later

[languid canyon]

how does linux solve it

[fallen bobcat]

I think Linux has per-PTE locks which they might be using here

[languid canyon]

btw on linux the entire vmalloc area page tables are pre-populated

[fallen bobcat]

Linux's memory management isn't the greatest so I'm not taking everything its doing as a good thing

Yea I suppose that could be a solution but my heap space is like 1 TiB so idk how you wanna preallocate that

[languid canyon]

linux vmalloc area is 32TB

or actually, only the top level pages are preallocated

lower levels are lazily populated and are guarded by mm->page_table_lock

https://elixir.bootlin.com/linux/v7.1.2/source/mm/memory.c#L6817

i guess they never found this lock to be particularly contended so it still uses the global mm lock lol

oh ok so level 2 page tables actually do get a spinlock that lives inside the struct page of the parent

https://elixir.bootlin.com/linux/v7.1.2/source/include/linux/mm.h#L3832

and pte too interesting https://elixir.bootlin.com/linux/v7.1.2/source/mm/Kconfig#L554

[fallen bobcat]

I think I will have to drop UM

the way I want to do stuff kinda requires manipulating PTEs directly and I don't wanna deal with that

[languid canyon]

Lol

Weren't you praising it

[fallen bobcat]

it's good for early development

for vmm stuff it could be done but idk how

[languid canyon]

So are you doing per pte spinlocks like linux

[fallen bobcat]

I'll do this for now

drop the lock, allocate, grab the lock and bail out if needed

[languid canyon]

Yeah but where does that lock live

[fallen bobcat]

then in the meantime I'll do generic kernel stuff and read about MMs

[fallen bobcat]

in the space

[languid canyon]

What

[fallen bobcat]

i call the thing every process has a space

I have one per space lock

[languid canyon]

So basically global page table lock like linux

[fallen bobcat]

sure

[fallen bobcat]

eventually it'll be broken down most likely though, I just need to know more about mms first

[languid canyon]

Or well, linux splits it for level 2 and per (user) pte now

[fallen bobcat]

what NT does is it checks if there are enough available pages for the worst case first

then does it all under the PFN lock

which I don't really like

[languid canyon]

But why do that, worst case is u have a useless allocation you free right away?

[fallen bobcat]

yeah

but NT also does more aggressive quota stuff

id have to relook at that code sec

[languid canyon]

Also how is that not subject to toctou?

[fallen bobcat]

because the pfn lock is held the whole time

[languid canyon]

Does pfn lock also lock the physical allocator?

[fallen bobcat]

the pfn lock is the physical allocator lock

[languid canyon]

Wait

Yeah the way you describe it just sounds kinda insane, maybe I misunderstood it

[fallen bobcat]

actually it does that for paged pool only

for nonpaged pool it's threaded through PTEs already

[languid canyon]

Is this leaked code or smth

[fallen bobcat]

it's wrk on a local elixir instance

[languid canyon]

wrk?

[fallen bobcat]

what do you not understand?

[fallen bobcat]

windows research kernel, NT code microsoft gave to unis

so it's kinda a leak but not really

[languid canyon]

Basically the way you worded sounds like it takes the global page allocator lock to populate every pte

But also that code is old af and concurrency requirements have grown a lot over the last few decades

[fallen bobcat]

yeah you have to do that

because the page count is protected by that lock

[fallen bobcat]

yeah now they have per-pfn locks

[languid canyon]

Does it check like every page table level and add that up and then request the total number or smth?

[languid canyon]

Ah ok

Seems like everyone does that now

Maybe ill start building with that in mind

How hard is it to roll local elixir

[fallen bobcat]

not sure, that code is tricky but broadly yes

[violet zenith]

Well, essentially, this is a violation of the license

[fallen bobcat]

that code has been on github for years and microsoft hasn't removed it

[languid canyon]

I wonder why they dont just open source it or at least make it source available

[rocky yoke]

just do fully lockless PTs as on Managarm

[fallen bobcat]

well, this is source available

but to unis

[languid canyon]

I mean win 11

[fallen bobcat]

i feel better about that rather than reading actual leaks

[fallen bobcat]

not that hard, i do it through docker

[languid canyon]

Like they could really elevate their reputation if they made nt public and allowed patches

At least the bare kernel

[fallen bobcat]

how do you solve the thing i brought up

[fallen bobcat]

this

[fallen bobcat]

the windows people we asked that to said that it's to avoid driver people relying on internal behavior

[violet zenith]

It does not mean microsoft granted a public license

The license in the repo is still restrictive

[fallen bobcat]

I'd have to read it again

[violet zenith]

It says rights are only available while the user is eligible through an accredited educational institution

And it allows public publication only of small teaching/research snippets, with each snippet not exceeding 50 lines, full distribution is limited to other eligible users

[rocky yoke]

by allowing the race

i.e. two threads will allocate the PT, one will throw it away

after seeing that it has been concurrently allocated

[fallen bobcat]

yeah but do you atomically update/read the PT?

[rocky yoke]

yes

[fallen bobcat]

huh

[rocky yoke]

that is essentially needed anyway

[fallen bobcat]

and you use rcu to free the pts

[rocky yoke]

yes

[fallen bobcat]

i guess i could do this

[rocky yoke]

the atomic update is needed in lots of cases anyway since you're racing with hw updates of AD bits

[fallen bobcat]

the way I'm designing SMR it will be tricky to use for larger allocations like that though

maybe i will have to do boring QSBR RCU

[fallen bobcat]

technically that website is unreachable, so I can't check eligibility

but honestly I don't really care about the license esp. considering I am using it for studying (its intended use) and not redistributing it anyway, and this is old software

if i share code however I could put spoilers on it to avoid people under employment (or working on e.g reactOS) tainting themselves

[languid canyon]

me when i recreate NT 1 for 1 after looking at 5 lines of educational xp from 2003

[fallen bobcat]

yea, obviously I try to avoid sharing code as much as possible but that stuff I didn't understand fully so I just sent it

[crimson sand]

i wonder how its done nowadays

as it doesnt make sense to me that uses per pfn lock

[fallen bobcat]

they could be locking the pfns of the ptes like linux

and do the retry thing

[solid marsh]

let me understand exactly the problem, you want to make sure that if more than 1 thread has to allocate a page table, only one succeeds, without using any higher order locks, and dropping the lock when retrying on oom?

[fallen bobcat]

well, to map a page you might need to allocate PTEs, and allocating blocks so you need to drop your lock (in order to avoid a deadlock where you would want to free memory from the same space) before blocking waiting on free memory, then when you wake back up, you re-take the lock but that PTE might've been allocated by someone else in the meantime. Basically:

lock.acquire();
if !pte.present:
  lock.release();
  page = alloc_page_for_pte();
  lock.acquire();
  pte = page; // oops, someone else might've done this already 

so the idea is to either 1. lock the pte somehow or 2. just re-check upon wakeup and bail out

[solid marsh]

isn't it possible to use a compare and exchange?

allocate page, read pte, if present is set, go out, otherwise, compare and exchange

read pte <-> compare and exchange in a loop

[fallen bobcat]

yeah that is what korona proposed i think

but comparing it after taking the lock is fine here too

[solid marsh]

how do you make sure the page table where the pte is located doesn't disappear?

[fallen bobcat]

the lock would protect against that

[solid marsh]

okay, but you release it to allocate the page

[fallen bobcat]

oh that

yeah

uhh idk, good point

i kinda don't wanna do lockfree stuff yet because I don't have an RCU mechanism (yet) and I don't have the broad overview needed to not design myself in a corner

I guess one way would be to have one deletion lock you take across the entire operation and that lock is only taken during deletions

but then it could block indefinitely on waiting for memory if it can't delete

that operation should be canceled anyway if it's getting destroyed

idk

[solid marsh]

maybe in the task struct you could store, to say, the virtual address range you're trying to allocate page tables for, and you link this struct in the mm_struct

the linked list could be protected by a spinlock, which is taken during free operations, which shouldn't block

when someone is freeing memory from your virtual address range, it could take that into account and not free the page tables that are to say, reclaimed to be mapped

[fallen bobcat]

kinda like hazard pointers

for now this isn't an issue btw since only the kernel heap exists

but it's still a thing I hadnt thought about

[solid marsh]

and that lock is only taken when:

• freeing virtual address ranges
• during the linking for this data to the list by the task, and during the unlinking

[languid canyon]

I think it could be solved much simpler

[fallen bobcat]

i still think that could be solved with pte locks

[languid canyon]

userspace PTEs are backed by vm areas, you can just check whether theres a vm area that overlaps this page table level and not free it

and obviously the vm area is locked at this point

[solid marsh]

this sounds like a more general version of my proposal

where the ptes are areas by themselves and not something special

[languid canyon]

Yeah

The invariant you get is you cant randomly have upper levels freed

[fallen bobcat]

couldnt you just take pte locks on all levels

[languid canyon]

Yes

But you have to drop it to allocate

Or you have to have a pre pass

[fallen bobcat]

no you don't

[languid canyon]

How

Unless your allocator never blocks

Which is a really bad idea

[fallen bobcat]

well

i guess it could work but it could still deadlock

[languid canyon]

Yes, you need the pte locks to evict memory mapped files etc

If you happen to run low during this page fault etc

[fallen bobcat]

well just freeing memory close to that address would require taking the same locks

[languid canyon]

Yeah, you also said taking all level locks

Which essentially means locking the entire address space

[solid marsh]

this is why i don't like memory management

[languid canyon]

so imo for userspace vm areas are required for this

[fallen bobcat]

for now I'll do the thing i said earlier because you cant delete the heap PTs

[languid canyon]

and for kernel u can get away with never freeing upper level PTEs

[languid canyon]

yeah thats what i do also

worst case is u hit the upper bound for your heap range

[fallen bobcat]

then I'll read mm books and I will come back with a solution hopefully

[languid canyon]

What are said books

[fallen bobcat]

rn i'm reading "What makes it page?"

[solid marsh]

that one

i have an amazing setup

win7 + win10 vm using libvirt and communicating with each other with serial

[fallen bobcat]

but also I'll read the freebsd book's part on this, and chuck cranor's phd dissertation

[fallen bobcat]

to play with the debugger?

[solid marsh]

god bless there is virtio 9p for win10, but i have to constantly insert and eject cds for win7

[solid marsh]

yeah

also, windows 7 internet connection is broken because all root certificates have expired, so i have a special python script for downloading the windbg data inside win7

[fallen bobcat]

no way I'm doing that

I don't have a windows vm setup at all

well i guess I could

dunno how useful it is

[solid marsh]

the book is very hands on

[fallen bobcat]

omg 😭

[solid marsh]

tbf, i have not read it all, i reached page 200

[fallen bobcat]

i skipped the earlier chapters because it's stuff I already knew

I skipped to chapter 20

[solid marsh]

oh yeah

chapters are more like subchapters lmao

[fallen bobcat]

yea

[solid marsh]

a fun issue i had with the windows 7 serial is that the kernel recognizes it during early boot for windbg server, but refuses to make it available for userspace afterwards

it's because the io space for the serial overlapped with the io space claimed by the chipset iirc

the device manager would show a ⚠️

[languid canyon]

Why does windows 7 need internet connection?

[languid canyon]

U can fix that in QEMU btw

Just use a different io port

[solid marsh]

i installed vc and the debug tools because i wanted to use live kernel debugging inside it

[solid marsh]

oh neat

[solid marsh]

the installers come as either self contained exes or isos, but the debug information has to be downloaded from the internet

[fallen bobcat]

man my page table abstraction sucks so bad 😭

[solid marsh]

can't be worse than mine

i delegate all to the arch code

😢

[fallen bobcat]

I don't do that but it's very weird

basically the pte stuff is abstracted away so it basically renders the generic pte manipulation code useless

this was because on UM it just uses mmap, but now I'm thinking of just nuking that code

[languid canyon]

...actually that might've already been patched in latest qemu

[rocky yoke]

Well, you could do that

It's not impossible

By moving all blocking out of the allocator

For example by using the magazine reload idea that i posted earlier

[fallen bobcat]

I'm not sure i understand that

typically you do blocking on the physical allocator level no?

[languid canyon]

maybe korona means u can pre-fill it before doing the thing

Like make sure you have some percpu memory ready to go to cover the worst case so no blocking is required

[fallen bobcat]

no way

nadia mention

[solid marsh]

lol

[rocky yoke]

yeah

[fallen bobcat]

do you mean magazines on the physical allocator? or what

[languid canyon]

Yeah

[languid canyon]

Where

[fallen bobcat]

search for babaoglu-joy on this server

but write it properly

yeah ok search for Babaoğlu-Joy

[languid canyon]

Oh

[fallen bobcat]

gotta love the hand drawn diagrams lol

it's impressive that a random guy wrote a 600 pages book on the windows 7 memory manager

[languid canyon]

where can I acquire it

[fallen bobcat]

if you can afford it I think you can buy it on amazon

i found it on anna's archive but this kinda content is made by a (afaik) hobbyist and it'd be nice to support him

[languid canyon]

yeah amazon definitely ships stuff here

Did that guy work at microsoft?

[fallen bobcat]

I don't think so

[solid marsh]

nl

[fallen bobcat]

do you have the paperback?

[solid marsh]

yes

[fallen bobcat]

nice

id buy paper books if i had space for them

(and money, though in this case it's pretty cheap)

[vernal oriole]

If you want I have a pdf

I could dm it to you

[languid canyon]

nah dw

[fallen bobcat]

This thread is now the 🏴‍☠️ thread

[long pendant]

after getting home i'm afriad i do not remember more

not much more anyway

what i can say for sure is that there are benefits to having a distinct creation/page table geometry lock from a stealing llock

[fallen bobcat]

How do you handle the case where the page table is getting deleted while you're waiting to allocate memory to fill it?

[long pendant]

i pin them with refcount biases as i descend them

[fallen bobcat]

Ah so when you're descending ptes you reference them

[long pendant]

but locking the creation locks (which are intended to protect page table geometry) would also suffice (and that is how i can allocate, which may take others' stealing locks, while still holding the creation lock - holding of creation locks establishes important rules i rely on)

[fallen bobcat]

And then you dereference them at the end of the operation?

[long pendant]

yes but that took me a second

[fallen bobcat]

ok I see

[long pendant]

i can't stand that terminology 'dereference'

i know it's traditional to windows but to me dereferencing is what you do to a pointer

[fallen bobcat]

I like retain/release too

is the reference count updated atomically and sits in the page struct? I'm wondering if i can make this all lockless like managarm

[long pendant]

it's nonatomic and guarded by the per-vm-domain page queues lock

(it's nukeyronex that i am describing)

but i do have an atomic field

it's a polymorphic field

for page tables it's a nonzero PTEs count, for other things it's a share count, and while nonzero it contributes 1 to the reference count

so only when that count (which is atomic) would transition 0 to 1 or vice versa do you need to lock the page queues lock

this can probably be synthesised with hyena's work on individually locked page queues

[fallen bobcat]

Huh the pte refcnt is protected by the list lock?

Interesting

[long pendant]

the refcnt is found for other things too

for all pages even

it's the state of the page with respect to the page allocator basically

with respect to the per-VM-domain allocator i should say

[fallen bobcat]

Ah yeah I see

I'm not doing NUMA because it scares me

[long pendant]

i had been working on doing some semi-formalisation of the VM principles in nukeyronex

[fallen bobcat]

Unrelated but one thing I wanna steal from mintia too are continuations

[long pendant]

what i have with the creation lock, according to my notes, is the following: a guarantee that zero PTEs stay zero, valid PTEs stay valid, PDEs whose table contains >= 1 PTE that prevents swap of that table, stay table PDEs

what transitions are allowed?

• the page thief can turn standby into swap, standby-pde into swap-pde, and this may trigger demotion cascades (if a page table contains only swap PTEs then the PDE referring to that table can become standby)
• busy PTEs can be resolved to either valid or zero PTEs
• hardware updated A/D bits

this is useful because:

• you need to drop stealing-locks to allocate pages (allocator can acquire stealing-locks to steal a standby page at point of need)
• but with the rule i give above, you do not always need to re-load and re-classify the PTE you want to do something with; it cannot change from valid to zero or zero to valid

so that rule can considerably simplify code

i will try to write up what i've noted at some point in the near future

[fallen bobcat]

If I understand correctly, you can't delete pages from a page table if the creation lock is held, correct? Then couldn't you deadlock on waiting for free pages

Or I guess only the pages filling the PTEs can be destroyed (the ones that are mapped) but not the PTs themselves

[long pendant]

i defined a global lock class order and sketched out every codepath (every kind of fault, everything the WS ager does, everything the page stealing logic will do, everything the the modified page writer does, munmap, fork, pagein resolution, etc) and i think i have proven each case follows the global class order and is thus free of any possible deadlock

[long pendant]

you cannot turn a valid PTE into any kind of invalid one; a standby PTE can be turned into a zero one under the creation lock, though

the only guarantees made are valid stays valid, zero stays zero

[fallen bobcat]

but then how can you avoid deadlocks? For example, if you hold the creation lock across the wait for free memory, then someone else wants to free their pages but cant since the creation lock is held

or perhaps i misunderstood and you can free memory without holding that lock

or you release the lock before the block, and the ref count will keep the ptes wired ig

[long pendant]

oh, there's no holding while waiting for free memory

that causes back-out from the page fault and waiting on available memory event

but that is hopefully rare since aging is meant to keep a balance of standby pages to active ones, and pages can be stolen at the point of allocation

[fallen bobcat]

yeah, I'm testing out low-memory scenarios without paging

[fallen bobcat]

ok and the thing that protects the upper levels from getting freed in the meantime is the refcount? I think I get it

and that refcount is propagated? You would have a refcnt of N in the topmost pml4 entry if there are N operations referencing memory under it?

[long pendant]

it sort of emerges naturally, if a deeper-nested page table has nonzero PTEs count (what i pin a reference on since this does contribute a refcount of 1 while > 0) > 0 then there must be a valid PTE referring to it in the next-closer-to-top page table

[fallen bobcat]

Yeah but you'd need to wait until there's a reference count of 0 on the pml4 before being able to reclaim it

idk I'll have to rethink about all of this maybe I'm getting confused

[long pendant]

i'll come back around to thinking about it in depth sometime in the near future

per-page-queue-locks and finer locks on page tables/object tables is just too compelling

but that time of the year when interest cycles to another project came round for me so i am working on my language at the moment (for writing a kernel in, of course)

[fallen bobcat]

do you have resources other than what makes it page and the UVM dissertation to learn about all this? I think I know way more right now than even a few days ago, but i wanna go in depth

and even then the uvm stuff is more implementational

[long pendant]

vax/vms internals probably worth reading, the "Segmented FIFO" paper, a later paper called something like "Paging dynamics in Windows NT", i actually struggle to name any that deal in the area i think we are concerned about here though, which is that sort of middle-ground, where you're not dealing in the broad dynamics nor the subtleties of single lines of code, but the locking, synchronisation, lifetime, ownership, and other matters

i had to figure those out virtually from zero

[fallen bobcat]

I see

thanks

I think I'll start with coarser locking then figure out how to smash the locks

i wanna try doing lock-free stuff as well here

[long pendant]

i think per-working-set, per-vm-object, etc sort of locking is a reasonably fine model to go by, since the heavy parts of fault handling etc happen with them unlocked

per-page-table is subtler

[fallen bobcat]

I think NT does it under the working set lock

from what I read in the book so far

[fallen bobcat]

that is, managing the page tables

[crimson sand]

i see MiLockPageTableInternal/MiUnlockPageTableInternal in the symbols as well

each process also has a PageTableCommitmentLock

[fallen bobcat]

i meant win7

idk modern NT

you should write what makes it page for win11

[fallen bobcat]

i feel like even with source code it would take a ton of time to understand

[fallen bobcat]

ok now MM abstractions are more sensible

I took inspiration from keyronex

[fallen bobcat]

mhm an interesting problem that could happen and stop the fireworks from continuing: multiple threads are trying to allocate stacks for threads at the same time, they could hold memory and block, deadlocking

like, thread A tries to allocate 4 pages for stacks, has already allocated 2 and blocks since thread B allocated 2 out of its 4 too, and is now blocked waiting for free pages

the solution would be to either make blocking for free memory transactional, or free pages for stacks if we need to block

[fallen bobcat]

uh what

ig the refcount keeps it wired but it's weird to phrase it that way

[fallen bobcat]

ngl idk if zig was the right move

but too late

what imma do next is doing proper zone allocator management

like, a worker thread will keep enough magazines using a working-set algorithm

and also reaping when low on memory

[languid canyon]

lol

why

[fallen bobcat]

idk i just like C a lot

may sound weird, but I think I like headers better than modules

actually first thing I'll do is nuke UM

@languid canyon you will be happy

[languid canyon]

yes

[fallen bobcat]

rip UM 2026-2026 🥀

[heavy sandal]

wait what is/was UM?

[fallen bobcat]

kernel running in usermode

[heavy sandal]

ah ok

[languid canyon]

Now do usermode running in kernel

[fallen bobcat]

it's done 😭

poor guy has been put down

jk idrc about it

tho it has been good to me

https://github.com/rdmsr/zag/commit/c90b327340f2c347b97c01814ceea713d552e998

[fallen bobcat]

i generated this cool logo using a script

[fallen bobcat]

annoying double iteration

illumos just lets it race lol

[rocky yoke]

why did you remove user mode?

[vital summit]

W

[drifting blade]

How

[fallen bobcat]

I like manipulating PTEs manually, and it is tricky to emulate

It could be done, but wasted effort on something that isn't that useful

[fallen bobcat]

ok very cool, now the zone allocator trims magazines periodically using a working set algorithm based on a weighted moving average

every 15 seconds it computes its working set size and WMA and then it decides whether or not to trim based on:

1. if excess empty mags > configured excess empty mags (8)
   or
2. if excess full mags >= 2 and excess full mags take up at least 16 KiB

solaris originally didn't use a WMA but instead just a minimum count (it also didn't trim periodically but only when memory is needed), I stole the heuristics from XNU and also do trimming periodically

[fallen bobcat]

oh also I did the linux watermark thing for my memory thresholds, I REd modern NT and it wasn't clear what they were doing, something like on smaller systems it's 1/16th of free memory, and on large systems it grows slower, at about 1/64th (very roughly)

so instead I do the min_free_kb = sqrt(pages * 16) or whatever the calculation is

[heavy sandal]

how often do you compute that? because sqrt is slow af compared to a shift-right division

[fallen bobcat]

once

you do it at boot

[heavy sandal]

cool

[fallen bobcat]

there's a cool oracle article on this

https://blogs.oracle.com/linux/anticipating-your-memory-needs

[heavy sandal]

i think this is correct btw, though the divisions might be different - NT has a bunch of memory stuff that changes if the total system memory is above a certain threshold

[fallen bobcat]

basically this

(yes I used codex with IDA pro fight me )

[heavy sandal]

this does match what ive seen except this is ternary hell instead of some nested ifs

[fallen bobcat]

shit i've been nerd sniped once again

i was gonna do magazine resizing but xnu does something cooler (and it seems better) so I'mma do that

[languid canyon]

What does it do?

[solid marsh]

magazinemaxxing

[fallen bobcat]

instead of resizing magazines it has a per-cpu depot

which increases in size to reduce contention on the zone depot

the traditional solaris thing is to resize the magazines

but this involves a bunch of extra mechanisms

https://github.com/apple/darwin-xnu/blob/main/osfmk/kern/zalloc.c#L298

@long pendant you might like this, it's more elegant than purging and resizing magazines imo

XNU and FreeBSD's allocators are a goldmine for modern slab allocators (XNU's is mostly based on freebsd's too afaik)

[languid canyon]

9k loc 💀

[fallen bobcat]

Yeah it's pretty crazy

They also do smr tho

And a bunch of security stuff

[crimson sand]

what function is this from

[fallen bobcat]

I forgor sorry 😭, I'll get back to you on this

[crimson sand]

kk then

[rocky yoke]

btw why are you going with magazines instead of sharding?

[fallen bobcat]

because I had them before I knew how to implement sharding

also I feel like this is easier to tune and manage than sharding

in mimalloc fashion you assume a regular allocation workload

ah also cross-cpu frees are more common than in userspace so that's annoying

I'd also have to think about how to SMR through that, with magazines I can just stamp the magazines with a sequence number and free a bunch of items at once

[rocky yoke]

that's true

[fallen bobcat]

idk, it is definitely an interesting area tho

[rocky yoke]

yeah i think both strategies are fine / comparable in performance etc when properly tuned

the trade-offs are a bit different ofc

[fallen bobcat]

you could probably get an edge with mimalloc-style sharding if you manage to allocate and free on the same CPU more often

idrk

[fallen bobcat]

i swear this file will end up 2000 lines

[fallen bobcat]

holy shit i am in so deep

[fallen bobcat]

https://github.com/rdmsr/zag/commit/a39f6a2f75978dfc2c042dc93ac8d10be8ee2331 finally done

idk if it works but i think it does

[solid marsh]

jesus

the file is

big

[long pendant]

that's smart

i like it

collecting actual data on contention and sizing accordingly seems the best approach by far

[steady swallow]

Zig kernels are tuff

Writing mine in Zig. C is well too fragmented

[fallen bobcat]

Fragmented?

[vital summit]

what does ts even mean 💀

[fallen bobcat]

Honestly I don't really care about zig as strongly as to advocate for it against e.g C

like, I wouldn't defend it as much as C in a language war

[heavy sandal]

I like zig and dislike c and I still can't figure out what that message means lmao

[vital summit]

fragmentation is when headers

[drifting blade]

If you drop a C it breaks and fragments into parts

[languid canyon]

They probably mean you split code between .c and .h

[fallen bobcat]

nah i think they mean the ecosystem/coding style?

like, C has tons of different implementations and people all write it differently

idk, it's not like I write conformant zig either (the "official" style sucks)

[drifting blade]

Why?

I enjoy lowercase _ seperated everything, if that's what you mean

[fallen bobcat]

I don't like the function names

i use snake case

they use uglyCase

[languid canyon]

Yeah JavaScript case sucks

[steady swallow]

I mean, it's not standard.

Like the types: long long Is a great example, wtf is that meant to mean

strtoul

[fallen bobcat]

C is more standardized than zig

[steady swallow]

Not really?
Zig has nice, cohesive types

Unlike C

[fallen bobcat]

zig does not have a standard, C does

[steady swallow]

Unless you use something like stdint.h

[steady swallow]

I'm not talking about the standard or not. I mean the C standard is weird

[fallen bobcat]

you said standard

[steady swallow]

Like the types, or the fact you can do things like i = i++

[languid canyon]

Why is that bad

Oh

[steady swallow]

Which one runs first? Who knows, only the compiler

[languid canyon]

Dont write code like that

[steady swallow]

And how the hell is anyone meant to know what ulong long means

[steady swallow]

Yeah but you can and that's the issue. In Zig you can't by design

[languid canyon]

Use stdint which all compilers have

[steady swallow]

Mhm. But it's just unclean. We shouldn't have to use a extra header for types. Zig has nice std types builtin

[languid canyon]

Idk, I don't need forced handholding when I can avoid it

[fallen bobcat]

i think zig types are cool

but consider that C is like

50 years old

also I am cooking up something rn it's gonna be very cool

[languid canyon]

Yet another allocator rewrite?

[heavy sandal]

gonna bust out the zig standard in-the-lang type c_ulonglong

[fallen bobcat]

i never rewrote the allocator, only improved it

[languid canyon]

Is it allocator related tho

[fallen bobcat]

nah

[languid canyon]

[fallen bobcat]

I will do SMR stuff soon ™, but reading the XNU code it scares me a bit

like it's the kinda stuff that could easily add 1k lines to the allocator so I'm not doing that yet

[solid marsh]

lol why

(why would it add 1k lines to the allocator)

[fallen bobcat]

it's pretty complex

I am doing very cursed zig stuff atm

I don't think I have done something this cursed with the type system yet

[heavy sandal]

👀

[fallen bobcat]

the comptime parent: type stuff is cursed

[heavy sandal]

you dont need is_directory to be a bool btw

can make it a u0

[fallen bobcat]

ooh

[heavy sandal]

since youre only checking for its presence

not its value

or even void tbh

any OPV type

[fallen bobcat]

OPV?

[heavy sandal]

one possible value

[fallen bobcat]

ah

[heavy sandal]

the OPV types i know of rn are:

• void (value is void itself)
• u0 (value is 0)
• i0 (value is 0 or -1 idr which)

oh and any struct with no fields or any enum backed by u0/i0 are OPV as well

values of OPV types get inlined to a comptime constant of the one value whenever actually used and take up 0 bits at runtime

which is why theyre useful here

(there is one bug i know of with them which is that because they get optimized out you cant use them as the type of extern vars/consts used for marker addresses like beginning and end of sections)

that reminds me that the zig people recently redefined bitcast (in a way where itll work the same but has a sane and standards-friendly definition) and in the process made a bunch of type system crap so much clearer lol

[heavy sandal]

anyway, whats this for?

[fallen bobcat]

something kinda like sysctl ig