#MINTIA (not vibecoded)

1 messages · Page 14 of 1

twilit smelt
#

and stuff

#

which is not the same as the modified page writer

#

they have both

#

for reasons

#

at least last i checked

#

is it landy wang

#

ik he's mm but i know people start wearing multiple hats when theyre around that long

#

i see

mortal thunder
digital pivot
#

do you know if they changed how mm scans working sets ?

twilit smelt
#

hes someone id like to meet one day if AI-fication doesnt take all the jobs first

#

much has come down to us about landy wang's mm work lol

#

we actually figured that out

#

some time ago

#

based on changed structures in recent versions

#

or @warm pine did

#

he might like to talk to you about that

#

afaik he figured this out just from looking at changed structure definitions

#

lol

#

nice to see you confirm his hypothesis

#

i think he decided on doing it the same way in his hobby kernel keyronex when he gets there

#

theyre from the adjacent private server i mentioned

#

which is my project's server

warm pine
#

But the names were very clueing

#

Then the algorithm is implied by the structure itself

#

Since you only have page tables linked onto your queue, it is those you must be scanning- presumably all their contained PTEs and you can age them

twilit smelt
#

iterating 512 times per page table that might only have a few valid PTEs sounds bad

#

i guess you could look up the first mapped region that lies within the page table and start there and scan by region

#

but now your lock for your mapping structures is subordinate to a memory-making lock (the working set lock)

#

and now you cant put them in paged pool and stuff

#

:(

warm pine
#

Looking up map entries less nice

twilit smelt
warm pine
#

And one assumes the typical case is you usually have more than a few valid ptes

twilit smelt
#

the high priority working set trimmer thread iterating over just one page table for each of like say 10 processes could cause a noticeable UI stutter

#

at 20mhz

#

if that was 1024 iterations each

warm pine
#

And on the software side you expect local blocks of data accessed frequently to have also scaled up commensurately, with isolated 4k blocks probably much less common

twilit smelt
#

there has to be a way to accelerate it

#

if i used a bitmap that would introduce a 1024 / 8 = 128 byte overhead for each page table but would let me skip groups of 32 ptes at a time

#

oh

#

i was already planning on having bitmaps though

#

for commit tracking

#

so i could probably reuse that

#

since only a committed area can have valid PTEs

#

mm nvm thats not applicable

#

that was specifically for tracking page table commitment

#

like charging commit for the maximal number of page tables that may need to be created to map an area

#

and instead of a bitmap i was going to do something a bit different

#

for each level of the page table hierarchy except the leaf level i was going to have a distinct avl tree

#

per process

#

which are used to look up arrays of counts of mappings that overlap each lower-level page table

#

when it transitions from 0->1 the commit for that page table is charged

#

1->0 its uncharged

digital pivot
#

and they left too

twilit smelt
#

shame

digital pivot
#

yeah, wanted to reread some messages because i didn't fully understand them yesterday but theyre not there anymore

twilit smelt
#

feels unfortunate that industry people aren't comfortable discussing their work with aspiring people

digital pivot
#

damn, we'll never know what cache manager problems and design limitations they were referring to

warm pine
#

shame, seemingly they just came to see if any of us knew that paper about linux

twilit smelt
#

the other NT guy we had in here was also quite skittish and would often delete messages and leave so i get the impression these people are terrified of HR

warm pine
#

i was in a hurry to give a response at the time but now that i am home i can be clearer about what i meant

digital pivot
#

there's a ms agent sitting here, and if they see any ms developers talking about their work, make them leave

night needle
#

hr is scary

twilit smelt
#

Yeah

#

Not saying it isnt

#

they also might be afraid of who would be attracted to a legit NT kernel guy who is willingly conversant in some public internet space

#

Skids galore if the wrong people heard

night needle
#

oh ya

warm pine
#

basically i am taking as a given that: in 1989, it was a common pattern to have individual 4k pages sparsely populated in the address space, while in 2025, you are much more like to have at least dozens of 4k pages living next to each other and acting like a unit, either all of them being active or none of them

digital pivot
twilit smelt
# warm pine this is an interesting case since here you see how the changes in the scale of s...

I thought of a possible solution which is that when a page table is created for the first time, I can look up all of the map entries that overlap the region mapped by the page table and create smaller "mirror map entries" in nonpaged pool which just represent extents of PTEs that may exist in the page table. These would be threaded along a list anchored on the PFN entry for the page table

warm pine
#

that's one half of why i don't think scanning a whole 512 PTEs at once is so bad; the other half is simply that scanning 512 PTEs is extremely fast on a modern machine, the access pattern involved is an ideal case, this is part of why there are so many anti-linked-list people nowadays

twilit smelt
#

So then I only need to scan the PTEs that are represented by the mirror map entries

#

If I don't have room in the PFN entry for the list head i can also just do like a separate map entry tree with small nonpaged map entries that function similarly, or split the normal map entries into a nonpaged and paged part and just use those.

warm pine
twilit smelt
#

They aren't that big

#

they were only 52 bytes apiece in old mintia

#

or 64 bytes of pool

#

counting the pool header

warm pine
# twilit smelt They aren't *that* big

i know why you didn't like it in principle in the past, since it brings in a non paged factor which scales in a way that really pure virtual memory (which scales independently and doesn't produce a scaled cost in physical memory) should not, but i think there's no getting away from some impurity there

twilit smelt
#

yeah

#

i am doing disposable page tables this time though

#

rather than pageable page tables

#

so when the last valid PTE is trimmed from a page table it will just instantly go away

#

and page tables arent tracked in the working set

#

the info for where anon pages was swapped out will go in a vm object or amap like in UVM

warm pine
#

not for the fun of it though i do prefer it in abstract/aesthetic ways

twilit smelt
#

its more portable for one

#

i also think its probably more memory efficient

#

and more efficient in general

#

you cant have a fully abstracted pmap layer without the page tables being disposable

#

if you rely on them being a certain way then suddenly ppc gets more difficult and stuff

warm pine
#

for the practical reason that i want to support aging the working set, and on arches like risc-v, i really don't want to hold a spinlock to do this

#

i want to be able to do a sequence of atomic operations instead that should ultimately get me to setting the A-bit

#

and the approach i have in mind involves deferring the reuse of hardware page table pages for any other sort of page, in such a way as reuse can't happen until after - any page fault handler that began before we got rid of the parent PTE referring to that page - has completed

#

or rather that the "lockless A-bit/D-bit setting" part of a page fault has completed first

warm pine
twilit smelt
#

i have completely discarded that portability aspect already though lol

#

so im not doing it for that reason but rather for efficiency

#

i think having the page tables be disposable and having the "prototype page tables" for vm objects and amaps go in a special pinnable paged pool instead will be much more efficient

warm pine
#

in practice it might be that doing these is useful anyway for reasons of performance, and this would also get in the way of doing the new style approach to working set aging, but if you say working set aging is in the pmap layer (which is a division i think is reasonable) then you can feel better about it all

twilit smelt
#

ill probably do the cool pte scanning working set

warm pine
#

it's too cool not to do

twilit smelt
#

i dont have an A bit but there are two alternatives:

  1. simulating it w/ the page fault handler by toggling the valid bit off and using one of the available bits in the valid PTE as your A bit
  2. using presence of the virtual page in the TLB as your "A bit"
#

both have been used on A-bit-less RISCs in the past

warm pine
#

i want to support aging in the working set for improved replacement dynamics and this is clearly a superior approach to that, than the traditional WSL where you have the WSL then you have to look out-of-line, all-over-the-place to see the actual PTE

twilit smelt
#

also VMS used option #1 on VAX which didnt have an A bit

#

and probably also on Alpha which lacked one as well

warm pine
twilit smelt
#

because you dont need to manage this other structure

#

that needs to be able to grow and so on

#

its folded into page table management

#

that you already have

#

you also dont need to explicitly insert virtual pages into the working set

#

just by creating the valid PTE you have accomplished that

mortal thunder
#

my question is how do you "age" PTEs afterwards

#

where do you store the age of the PTE

twilit smelt
mortal thunder
#

on 64-bit its okay because theres like 16 spare bits (if you don't use the PTE key feature)

warm pine
mortal thunder
#

but 32-bit has like... three

#

or four

twilit smelt
warm pine
mortal thunder
#

but there may be other bits you want to use too

twilit smelt
#

and those are the only 32 bit architectures im supporting

#

ever

#

the other planned ISAs are Aphelion and AMD64 which are both 64 bit

mortal thunder
#

okay

warm pine
#

on RISC-V i think i only have 2 bits available so i have even fewer ages, but still you do get a few ages out of that, and that should be helpful

mortal thunder
#

worth noting that there is an A bit on x86 32

#

but there are 3 spare bits and surely you need some for something

#

in boron one of the pte bits tells me whether a page is mapped from physical RAM or is MMIO

twilit smelt
#

btw new boards of canada album soon probably

twilit smelt
#

you can get that info from the map entry instead

warm pine
twilit smelt
warm pine
#

there are probably other optimisations possible if you have more bits free but none that i see as really essential

twilit smelt
#

multics used page aging and they found they only needed like two page ages

#

the benefits dropped off DRASTICALLY after that

#

and at about 8 ages it started performing worse

#

this is about that

warm pine
#

one nicety i don't have a full vision of how to do yet - parallelised working set aging - probably once you already divide a working set into a list head per, say, average age of PTE within the page tables headed by that list, you can then individually lock these heads and do some magic to allow you to unlock the list head you were working on before and re-queue the page to another head (when i last thought about this, i expected some complication here, but now that i'm thinking of it again, i wonder if i was just blinkered by something stupid at the time)

twilit smelt
#

also possibly one of the earliest descriptions of clock

#

k is the number of ages

warm pine
#

and with that parallellised aging i think you alter the considerations a bit and more ages = more better up to some limit more like how many cores do you have? and how big is the program?

twilit smelt
#

smth i dislike about using the valid bit for page aging though is the number of tlb shootdowns it requires

#

probably you would just not do any shootdowns (they arent "functionally" necessary) while doing aging and then at the end you do a big shootdown that makes everybody dump their entire TLB

twilit smelt
#

and theyre individually locked on that basis

warm pine
twilit smelt
#

also are you excited for the potentially imminent new boards of canada album

warm pine
#

it seems believable as kelower said something to this effect once

twilit smelt
#

you can unlink it from the one you have locked

#

after youre done aging the PTEs within

#

and add it to a list of page tables that need to be re-linked into different age lists

#

anchored on your kernel stack or smth

#

and then when youre done you can go through and lock each of the age lists and insert the page tables into the new one they need to go into

#

something like that

warm pine
twilit smelt
#

they are scottish

#

despite the nam

#

e

warm pine
#

they are scottish apparently so they must be good

twilit smelt
#

they grew up in canada though

warm pine
twilit smelt
#

hence the name

warm pine
#

wtf they're from Cullen

#

i've been to cullen

twilit smelt
#

because its not functionally necessary for it to be on the correct one

warm pine
#

they're most well known for a soup of smoked haddock, it's got an unfortunate name

twilit smelt
#

i found out just recently that they once promoted an album by doing an ARG that involved a fake DCL command prompt on OpenVMS which was very awesome to learn lol

#

hm i thought itd embed it

warm pine
#

i had in mind something like after each table is aged, unhook it, release that list lock, then re-queue it on the right table, but it is better to batch this process

twilit smelt
warm pine
#

that it's perfectly fine to tolerate the tables being on the "wrong" list makes life easier

twilit smelt
#

i take destination+source ready queue locks (in address order), take as many threads off the source queue as necessary to reach balance, unlock the source queue, enqueue to the destination, then unlock the destination

#

the wrinkle is that i need to take each thread spinlock before enqueuing it to the destination

#

which is out of order with respect to its ready queue spinlock

#

so i just trylock the thread while holding the destination queue spinlock and re-enqueue as many threads as i can that way

#

and any threads that i couldnt immediately enqueue, i put in an array and do it individually at the end

#

so im trying to maximize the odds that im actually fixing an imbalance and not making it worse

#

due to something changing in the middle

#

this actually did make the fireworks test run visibly more smoothly with lots of processors which was delightful !

twilit smelt
#

the working set trimmers probably take some stuff shared that is taken exclusive in the page fault handler or other places

#

shared locking scares me because it is opaque to priority inheritance but its not actually an issue here because the working set trimmers are the only shared lockers and are already high priority system threads

#

when i think about implementing this i think of probably having a big page table tree lock

#

per process

#

which is taken shared by the working set trimmers

#

exclusive in page fault codepath

twilit smelt
#

while holding this paging tree lock shared

#

i guess itd be cooler if i could figure out a way to have per-page-table locking

#

oh well im still pretty far from doing any serious vmm work cuz i need my IO system working first according to the roadmap i set myself like 1.5 yrs ago

# 
Development is organized into base levels; initial development is done on UP and MP XR/station and at the end of each base level anything architecture-specific is brought up to fox32 and tests are written.

Executive base levels:
0 - Initial development
1 - Full loader, full scheduler and core functionality, pushlocks (Ke)
2 - Pool management (partial Mm), full object manager (Ob) and partial namespace (Ns)
3 - Object-level process & job management (Ps), IO manager (Io)
4 - Finished XR/station and fox32 system drivers
5 - Pagefiles, memory mapping, page fault handling, etc (full Mm)
6 - File viewcache (Fv) and filesystem drivers for AisixFS, FAT12/16/32, finished namespace
7 - Duplex driver (Du), executive pty console (Ec), and first userspace execution
8 - IPC

Userspace base levels:
0 - Initial development (executive base level 7)
1 - Basic uexec.dll functionality (dynamic linking, apc/signal dispatch, etc), SystemInit.exe starts
- Executive base level 8 -
2 - SystemInit performs basic system initialization (Si), Authority management server performs session control & logon (Ams)
3 - Mintia Command Language exists as a generalized DLL and parses and executes programs (Mcl), has a shell wrapper cmd.exe
4 - Trivial tools such as cat, ps, ls, kill, etc
5 - Management tools such as users, groups, etc
6 - Port toolchain, self-hosting system
#

im working on BL 3 (so im on BL 2)

#

Ps is done (has been for months) and Io probably would have been as well if i didnt spontaneously decide it has to be plug n play which i wasnt sure how to do and has required a lot of thinking

#

original idea was to just straight rewrite old mintia's io system, maybe would have been better

#

@warm pine OH

#

i just figured it out

#

you dont have to hold the age list lock that long

#

you can take it to pop a page table off the list

#

then unlock it

#

and crunch on that page table in parallel

#

and re-insert it afterwards

warm pine
#

i wonder where the other operations we need to deal with tables on the list come in

#

say in taking a page fault, you find the table you want to work on

twilit smelt
#

you can probably do something like this:

DeferredAgeList: AgeList[NUM_AGES]

take Process.PageTableLock shared

for Age = 0 < NUM_AGES:
    while true:
        take Process.AgeLists[Age].Lock exclusive
        PageTable = pop Process.AgeLists[Age]
        release Process.AgeLists[Age].Lock

        if PageTable == null:
            break

        NewAge = age/trim PageTable

        push PageTable to DeferredAgeList[NewAge]
    end
end

for Age = 0 < NUM_AGES:
    take Process.AgeLists[Age].Lock exclusive
    while DeferredAgeList[Age] isn't empty:
      push (pop DeferredAgeList[Age]) to Process.AgeLists[Age]
    release Process.AgeLists[Age].Lock
end

release process.PageTableLock
warm pine
twilit smelt
#

yeah you also might batch the popping

#

like grab 2 page tables at a time instead of 1

#

or something

#

probably dont want to take too many more than that

#

4 at most

#

this is probably also the only time the per-process PageTableLock is ever taken shared anywhere in the kernel

#

in my idea for how it would work

#

it would be taken exclusive in the page fault handler to hold the page table structure in place while a transition PTE is set up

warm pine
twilit smelt
#

so itd be basically analogous to the working set lock of old mintia

warm pine
#

the aging is part of the general case of trimming/aging so that's one major family of operations. then there are page faults, which are a bit like another family of operations, and we can probably see things like unmapping a region as being in the same family as page faults, in terms of how it interacts with the page aging process

#

and figuring out how the two intersect is the more challenging thing

twilit smelt
#

i feel like i have things pretty much figured out

#

id like to start working on it now but i really need the IO system first

warm pine
twilit smelt
#

oh yeah i totally forgot to respond to your dms about that

warm pine
#

it's frustrating as now my interest is cycling back to kernel

twilit smelt
#

now u know why i was constantly bitching about being bored while working on jackal

#

also i might end up just allocating a bitmap of valid PTEs, for each page table page

#

for 1024 entries it ends up being 1024/8=128 bytes

#

which increases the memory overhead of the page tables by about 3%

#

which isnt actually that bad?

mortal thunder
#

Where do you point to that 128 byte block

twilit smelt
#

PFN element

#

of the page table

twilit smelt
#

worst case number of scanning iterations for both before finding PTEs to age is 128/4=32 and 64/8=8 respectively

#

which makes it acceptable

twilit smelt
#

this shit is gonna be unimplementable

warm pine
#

multics, mica, ozix

twilit smelt
#

yeah lol

warm pine
#

very arrogant of us to presume we can succeed where the giants failed

twilit smelt
#

well we have 20/20 hindsight on an extra like 30-60 years of kernel design that they didnt

#

so we actually might be able to

warm pine
#

but there's more practical experience to draw on, and now the modern kernels are now in most ways more complex than these would've been

warm pine
twilit smelt
#

yeah we have an overwhelming advantage compared to people who were trailblazing

#

its not comparable

#

thats one reason multics is so amazing to me

warm pine
#

it's easy to forget sometimes that early cultures had to develop over a very long time to gain the colour concepts they did

twilit smelt
#

it did a ton of shit there was literally zero hindsight on and succeeded on at LEAST 40% of it lol

warm pine
#

they start off with a dark/light distinction, red usually appears next (blood), green (fresh vegetation), etc

#

and no one today appreciates that they name the colours with words that developed through a millenia-long dialectical process where certain needs were practical drivers to repeatedly segment and name parts of the colour space

twilit smelt
#

i like how the word "Asia" comes from the hittite word "assuwa" which referred to specifically a small segment of northwestern anatolia

#

their backyard basically

warm pine
twilit smelt
#

the greeks directly west of anatolia knew of the east as "assuwa" and eventually it became the western word for the entire massive eastern landmass all the way out to japan

#

which is very silly

#

behold: the entirety of asia

image.png
#

(as that word referred to in about 1000 BC)

twilit smelt
#

i was so shocked to see they even had lazy symbol resolution with dynamic linking

warm pine
twilit smelt
#

except they did it with a "linkage exception" rather than a direct call to a dynamic linker

#

and the dynamic linker was in the "kernel"

warm pine
#

i know in america when you hear that someone is an "asian" i think the usual implication is they are from china or some other country in the sinosphere

twilit smelt
#

yeah we imagine china, japan, korea

warm pine
#

while here it means first and foremost the subcontinent

twilit smelt
#

the middle east being "west asia" would break the brains of most americans lol

warm pine
#

and on twitter i often saw american chuds talking about how it's george orwell 1984 oceania that, say, some grooming gang was described as "asian" here

#

"this is proof that they are trying to trick people by making them think they are chinese or koreans" that sort of thing

hybrid condor
#

the exclusive solo debate phase

twilit smelt
#

mysterious

hybrid condor
#

(Ik someone must've deleted their messages, but now it looks hilarious)

digital pivot
#

they didnt delete all the messages btw

#

perhaps they said too much here

twilit smelt
#

or perhaps i should do a CS bachelors and CE master's or something

#

i have a friend who did that

twilit smelt
mortal thunder
#

i chose a computer engineering faculty

hybrid condor
#

Here are two stories. I studied CS and started working while I was still a student. Being honest, my education didn't prepare me for my low-level work. Even though it was a really highly rated program, I don't think it was worth it, except for the good connections I made. My friend studied EE/CE and to this day believes that the knowledge and thinking he gained then still guide him. I think courses like microprocessor architecture or microcircuit design can be useful.

twilit smelt
#

okay cool

heady bobcat
twilit smelt
#

its still semi reasonable for me to pivot to CE so i might do that

twilit smelt
#

which apparently they regretted

heady bobcat
#

lmao

twilit smelt
#

or CE major math minor

#

thatd be funny

#

CE+math double major would also be funny

#

also doable if im willing to do an extra semester or two and take on the associated extra loan debt

twilit smelt
#

im genuinely interested in both of those topics

digital pivot
#

well yeah but he left

#

couldn't resist correcting

#

youre right

twilit smelt
#

the more i think about it the more im pretty sure CS is not the correct major and i made a small oopsie

#

there would be quite little wasted time if i switched rn because most of what ive done lies in the CE/CS overlap

#

and would work at least as electives for one or the other

digital pivot
#

why majoring in math ?

#

how it would be more useful than just cs

twilit smelt
#

because i think math people are smart and its beneficial for abstract thinking to get really good at math

#

im also interested in math

hybrid condor
twilit smelt
warm pine
twilit smelt
#

bit of diversity in my degree would be good

#

CS+math is too theoretical and doesnt reflect my low level interests so i feel like CE+math is better

hybrid condor
twilit smelt
warm pine
# hybrid condor I need enlightenment to remember what VM looks like in NT

essentially it started with there's two levels of replacement, the working set level and the global level. a process has a working set, this was historically represented as a FIFO queue of in-core pages within the process; the kernel tries to balance the size of working sets based on the fault patterns of each process. after a page is evicted from every working set to which it belongs, it goes onto either a dirty or a clean queue; dirty pages are gradually written back and make it to the clean queue, and a need for a page can be satisfied from the clean queue. this is slightly simplified but that's the thrust of it

hybrid condor
warm pine
#

and the actual form of the working set queue was actually more like a sort of ring buffer, not a list. but anyway, you have in this what is called the segmented-FIFO policy

twilit smelt
#

also this is a hot take or even blasphemy to STEM people but like i sincerely believe humanities classes are harder than STEM classes lol

#

5 upper level STEM courses looks way less scary to me than 5 humanities courses

#

there are always debates about whether STEM or humanities is harder on like twitter and its mostly STEM majors calling humanities majors perma-broke children playing with crayons and humanities majors calling STEM majors illiterate morons

#

i lowkey side with the latter

acoustic sparrow
warm pine
#

but later in NT they started doing aging of the pages within the working set instead, so instead of just trimming working sets to maintain balance, they scanned the working set queue, doing the typical techniques you'll be familiar with - reset the access bit, increment the age if it wasn't set. so the first level of page replacement, the working set level, is now no longer FIFO, but already a closer approximation to LRU

digital pivot
twilit smelt
#

thats how i know im not "really" smart

#

because i dont know much about humanities

#

i feel insecure if a humanities friend breaks out philosophy quotes and stuff

acoustic sparrow
warm pine
#

but the traditional working set queue list structure was a poor fit for this, as you have basically a ring buffer full of virtual page numbers + other state you fancy putting there, and when aging, you have to then go look at PTEs spread all over the place

twilit smelt
#

this is how it works now

warm pine
#

so (here is what i have guessed, but it seems at this point almost certain this is basically what actually does happen), instead of having this working set list, which was a fantastic data structure for the FIFO replacement approach, you instead thread the pfn db entries (vm_page_t to you) onto as many list heads per process as you have ages

#

and you do aging by iterating such a list, for each page-table page, you age all PTEs within it, and you use spare bits in the PTE itself to store the age

digital pivot
twilit smelt
#

potentially as many as there are cores or some other ratio

warm pine
twilit smelt
#

needs more RE'ing to determine exactly how it is

warm pine
#

historically philosophers treated mathematics and mathematicians were philosophers

hybrid condor
#

Did you figure this out based on changes in data structures?

twilit smelt
#

also we're not sure if the "age" of the page table (which age list it is threaded onto) is based on the average or oldest age of any pte in the table page

#

my gut feeling is that its the average

#

and the average is recalculated during each aging pass

warm pine
twilit smelt
# 
//0x100 bytes (sizeof)
struct _MMWSL_INSTANCE
{
    struct _MMPTE* PteResumePoint[3];                                       //0x0
    ULONG LastAccessClearingRemainder;                                      //0x18
    ULONG LastAgingRemainder;                                               //0x1c
    ULONGLONG LockedEntries;                                                //0x20
    struct _MI_WORKING_SET_PAGE_TABLE_AGE_LINKS ActivePageTableLinks[8];    //0x40
};
#

yeah there are 8 of those

#

as many as there are ages with the 3 available bits on amd64

warm pine
#

the name alone basically tells you everything, and when there are as many of these in each working set structure, as there are ages, then there's no doubt left

twilit smelt
#

MMWSL = memory manager working set list

twilit smelt
#

without prompting

warm pine
hybrid condor
warm pine
#

he was very cryptic about it, i couldn't tell if it was trying to talk simply to people he presumed wouldn't have the foggiest idea what he meant, or whether he was deliberately trying to not say much

twilit smelt
#

from public debug symbols

#

so it should be clean to look at

twilit smelt
digital pivot
twilit smelt
#

i feel like a few yrs ago i had a reputation for being really speedy with finishing things, and now im known as being fairly glacial, i think i got majorly depressed since then

twilit smelt
twilit smelt
#

as i used to

warm pine
twilit smelt
#

i feel like being a stem or humanities person (exclusively) and talking condescendingly about the ppl in the other camp is probably a sign of having low self awareness and being insecure

#

and if such people were as smart as they thought they were theyd respect and be curious about all of it

craggy spire
# 
struct _MMWSL_INSTANCE {
    _MMPTE* PteResumePoint[3];
    ULONG LastAccessClearingRemainder;
    ULONG LastAgingRemainder;
    ULONGLONG LockedEntries;
    _MI_WORKING_SET_HARD_LIMIT* HardLimit;
    ULONGLONG TrimmedPageCount;
    ULONG AttachedThreads;
    LONGLONG AddedLeafPages;
    _KGATE* ExitOutswapGate;
    _MI_WORKING_SET_PAGE_TABLE_AGE_LINKS ActivePageTableLinks[8];
};

struct _MI_WORKING_SET_HARD_LIMIT {
    _LIST_ENTRY Links;
    _MMSUPPORT_INSTANCE* Vm;
    _MI_AGING_FACTORS AgingFactors;
};

struct _MI_AGING_FACTORS {
    ULONGLONG AvailablePages[8];
    ULONGLONG AverageAvailValue;
    ULONG AvailableDeviationPosition;
};

struct _MI_WORKING_SET_CONTROL {
    _ETHREAD* WorkingSetThread[2];
    _LIST_ENTRY WorkingSetExpansionHead[3];
    _RTL_AVL_TREE HardLimitWalkers;
    _MI_WORKING_SET_CONFIGURATION Config;
    _MI_WORKING_SET_AGING_CONTROL AgeControl;
    _MI_WORKING_SET_TRIM_CONTROL TrimControl;
    _KEVENT Events[5];
    volatile LONG PartitionListLock;
    volatile CHAR WorkingSetInitialized;
    LONG LowPriorityPassNeeded;
};

struct _MI_WORKING_SET_AGING_CONTROL {
    UCHAR Active;
    UCHAR PeriodicAccessClearing;
    ULONGLONG TrimWhileAgingLowThreshold;
    ULONGLONG TrimWhileAgingHighThreshold;
    _RTL_AVL_TREE AgeListWalkers;
    ULONGLONG AgedPages;
    ULONGLONG AgingTime;
    ULONGLONG MaximumPtesAgePerSecond;
};
twilit smelt
#

okay so there are two working set trimmer threads

#

always

#

_RTL_AVL_TREE AgeListWalkers; is also scary

craggy spire
# 
struct _MI_WORKING_SET_TRIM_CONTROL {
    USHORT TrimStamp;
    ULONGLONG AvailableDeviationAverage;
    ULONGLONG AvailableDeviation[8];
    ULONG NumPasses[5];
    ULONG Delays;
    UCHAR HistogramIndex;
    _MI_WORKING_SET_TRIM_HISTOGRAM Histogram[16];
    ULONGLONG LowPriorityWsThreshold;
    ULONGLONG LastAvailable;
    ULONGLONG MinAvailableGoal;
    ULONGLONG IdealAvailableGoal;
    _MEMORY_PARTITION_TRIM_WHILE_AGING_STATE TrimWhileAgingState;
    volatile LONG ManyLowPriorityAgesExist;
    UCHAR TrimActive;
};

struct _MI_WORKING_SET_TRIM_HISTOGRAM {
    ULONGLONG StartTime;
    ULONGLONG ElapsedTime;
    ULONGLONG StartAvailablePages[2];
    ULONGLONG EndAvailablePages[2];
    ULONGLONG TrimGoal;
    ULONGLONG PagesTrimmed;
    _MI_WS_CLAIM_INFORMATION Claim;
    USHORT NumPasses;
    USHORT TrimStamp;
    MI_TRIM_REASON TrimReason;
};
#

I found them interesting, so I'll post them here.

twilit smelt
#

they are interesting

twilit smelt
#

in _MI_WORKING_SET_AGING_CONTROL

#

which is in _MI_WORKING_SET_CONTROL

#

which seems to be global (to a partition object)

#

@craggy spire can you confirm whether _MI_WORKING_SET_CONTROL is ultimately part of _MI_PARTITION

warm pine
#

they could've done just [8] of them but they didn't

craggy spire
twilit smelt
#

called it

warm pine
#

if we knew the key by which they're entered to the tree then we'd have an answer

twilit smelt
#

so they have a (semi)global AVL tree of "age list walkers"

digital pivot
#

can you share MI_TRIM_REASON ?

twilit smelt
craggy spire
# digital pivot can you share MI_TRIM_REASON ? 
enum MI_TRIM_REASON : LONG {
    TrimReasonNone = 0x0000,
    TrimReasonLowPages = 0x0001,
    TrimReasonSlowStandby = 0x0002,
    TrimReasonFastStandby = 0x0003,
    TrimReasonForce = 0x0004,
    TrimReasonForceSwaps = 0x0005,
    TrimReasonExternal = 0x0006,
    AgeReasonLowAvailable = 0x0007,
    AgeReasonSlowStandby = 0x0008,
    AgeReasonFastStandby = 0x0009,
    AgeReasonOwedStandby = 0x000a,
    AgeReasonLowClaim = 0x000b,
    AgeReasonRequiredMinimum = 0x000c,
    AgeReasonHardLimit = 0x000d,
    TrimReasonHardLimit = 0x000e,
    AgeReasonClearAccessed = 0x000f,
    ThreadFaultClusterAging = 0x0010,
    TrimReasonMaximum = 0x0011,
};
mortal thunder
#

i wonder if its dopamine addiction or some shit

twilit smelt
#

several times with old mintia i wrote 10-15k lines of debugged code in a single month

#

ive yet to do that once with mintia2

digital pivot
#

ThreadFaultClusterAging

#

i feel like the world needs updated version of what makes it page

twilit smelt
warm pine
twilit smelt
#

yeah probably

#

they might age those more aggressively

#

when they bring it in they might leave a bit in the PTE that says "brought in by a clustered fault" which is cleared if the A bit is ever seen accessed, but until it is, they age it by like 2 at a time or something

#

total speculation but would make some sense

warm pine
#

i would've probably just brought them in with an initial nonzero age, i wonder what sort of thing they're doing instead

twilit smelt
#

yeah you could do that as well

warm pine
#

presumably there is a good reason for not doing it that way and instead doing whatever this is - i would guess something like "so that you can more quickly age out clustered pages by this mechanism, instead of having to trawl through everything in a working set to get to them"

#

it may be that they found evidence, that having cluster-associated pages treated the same as pages you actually faulted on, is pessimal. that sounds like a reasonable thing to conclude

icy bridge
twilit smelt
#

only for pages brought in due to clustering

#

and only until theyre actually seen accessed

#

but fadanoid's idea is better

#

you could just set the initial age to 4 (out of 7)

#

or something

#

same effect

#

@warm pine do people usually reset the age to 0 if they see the A bit set or do they just decrement it or something

warm pine
#

since they are trying to approximate LRU

#

so any access = a recent use

twilit smelt
#

true

warm pine
#

but this is also why LRU-approximating isn't a god

#

since pure LRU is not necessarily ideal

twilit smelt
#

my intuition would be you should just decrement it

#

or halve the age

#

or something

#

because just cuz a page was accessed once doesnt mean it will be again any time soon

warm pine
twilit smelt
#

ok ill do that if i remember this conversation happened in like 7 months

#

when i implement this shit

#

i havent been keeping notes, i used to keep really good notes about old mintia

#

i had some breakthrough about the IO system that ive been frustrated i forgot about cuz i didnt write it down

#

cant remember what it was

warm pine
#

oh, here is an idea for you btw

#

i was just taking a look at linux "mglru" to see whether they do reset-to-0 youthing or something else

#

they are using bloom filters somewhere to decide on whether to scan particular PTEs

twilit smelt
#

ahhh

#

that makes sense

warm pine
#

if you want to cut down on looking at definitely-zero PTEs but without the cost of a full bitmap for every PTE that's an option

twilit smelt
#

i had to google bloom filter ngl but yeah that makes perfect sense

#

lol

#

im pretty sure ive literally done that before but didnt know what it was called

#

jesus

#

"core memory"

#

stone age wikipedia article

warm pine
#

i can't resist saying "in-core" to refer to pages in memory

#

it's just too short and useful

twilit smelt
#

ill probably use a counting bloom filter

#

that is superior to the bitmap yeah

#

i can do it simply by having each group of 16 ptes represented by a 4 bit counter or something

#

i scan them if the count is nonzero

#

and i can quickly skip big swathes by loading many of these counters at once and checking for zero

#

that works!

twilit smelt
#

nvm it doesnt

#

its only 32 bytes because 1024/16 = 64 counters, then divide by 2 (cuz two counters per byte) and its 32 bytes

#

i did the math wrong

twilit smelt
#

and it looked like a single word

#

incore

#

not in-core

#

same with "bread" i remember being confused what "bread" was

#

TERRIBLE name

#

even after i knew what it did i still couldnt figure out why it was named after a random food item

#

before i finally read it as "b-read" one day

#

the worst part is, its not like the unix ppl didnt have uppercase to work with

#

they did!

#

it could have been bRead

#

all along

twilit smelt
#

which is guarded by the page table lock

#

so i dont need to take anything else cuz ill already be holding that when i made a new page table

twilit smelt
mortal thunder
twilit smelt
#

i can load 4 bytes at a time which is 4*2=8 counters which is 8*16=128 ptes checked and skipped at once

twilit smelt
mortal thunder
#

did early C compilers have a character limit?

twilit smelt
#

no

#

however

#

the object file format only had 6 character symbols

#

anything beyond that would just be silently truncated

mortal thunder
#

ah so it would just be called buffer

#

and it could confuse with buffer_write etc which would also start with buffer

twilit smelt
twilit smelt
mortal thunder
#

damn

twilit smelt
#

they should have used XLO instead of a.out

warm pine
twilit smelt
#

inchresting

#

ill still experiment with halving the age rather than resetting to 0

#

i have a gut feeling thatll end up being superior

warm pine
#

they also do something with pages accessed through FDs (since they don't have a unified approach to page cache access) which i haven't tracked down yet but i suspect those also get rejuvenated to newborns

#

actually it looks more complicated, there is a PID controller involved

twilit smelt
#

tossing out ALL the age information if it was accessed ONCE seems a little bit silly

warm pine
twilit smelt
#

probably there should also be a "things are very bad." escape hatch flag where if a low memory situation has persisted for longer than X amount of time, despite some aging going on, you just start trimming everything in sight regardless of age

#

at least starting with the worst average age page tables first

#

idk how long X should be

#

probably a few seconds

#

maybe like 5-10 sec and it stays in "trim everything" mode for 1 second before resetting back to normal aging activity

#

in fact i think this is functionally necessary to guarantee memory being made eventually

#

you could conceivably have a bunch of very rude processes accessing a ton of pages in a loop just to stop anything from aging out ever

#

need to have a way to trim stuff anyway

#

mintia2 will return to page table scanning after 4 years

#

for about a month before i implemented working sets that was how i did swapout in old mintia

#

it also had a synchronous IO system and had an identity mapping and other stuff

#

it changed sooooo much

#

now i wonder if i can even build some random old version of old mintia

#

itll be fun cobbling together an archaic toolchain to try to do that i think

#

ok lets do it

#

ill do january 2, 2022 mintia

digital pivot
#

idk if its parallelized between multiple partitions

#

there probably has to be something like that if they can share memory

twilit smelt
#

ok i did it

image.png
#

very very old mintia

mortal thunder
#

feels weird seeing this again

#

i think nanoshell was in its very beginnings back when you were at this stage

twilit smelt
#

then for some reason i sent myself back to the stone age and yearn for mintia2 to be this functional again lol

#

omg... you can... enter commands...... and it spawns processes.... and they do stuff....

mortal thunder
#

i will need to do several things before i can do that including writing a pty driver

#

(it'll probably be located directly in the kernel though)

twilit smelt
#

lol current old mintia can still mount this old disk image but it cant run any of the programs

image.png
#

even the ISA changed a lot so im not surprised

acoustic sparrow
twilit smelt
acoustic sparrow
twilit smelt
#

or userspace

acoustic sparrow
#

just call it a microkernel and put off vmm for later(tm) meme

twilit smelt
#

i know exactly what to do, the issue is more mental energy

#

also the mintia2 plans are way more ambitious

#

when its done itll be substantially larger and more sophisticated than old mintia

#

by like 2-3x

#

which is intimidating because it took me 3 years the first time

mortal thunder
twilit smelt
#

thats true but mintia2 also has a lot of new stuff

mortal thunder
#

so the sophistication should cancel out with your amount of experience in an ideal world

twilit smelt
#

new to me

acoustic sparrow
#

i have a suggestion

twilit smelt
#

the vmm will be wayyyy different and the IO system is gonna do plug n play and fancy dirent cache stuff

acoustic sparrow
#

work on mintia2 instead of talking here

twilit smelt
#

does that count

#

most of my commits lately have just been from staring at code i already wrote nad going "hm thats slightly inefficient"

#

the actual next thing i need to do is initializing the iokit-like IO catalog

#

and probably fleshing that stuff out by just writing a driver

#

and implementing whatever i need

#

for driver matching and stuff

#

this would have been very exciting to me in like late 2022 but rn it sounds dreadfully boring

#

nowadays its almost good enough just to have designed it in my head and actually implementing it feels boring now

#

because i know itll work

#

theres no like excitement

#

or risk

#

i did this all before and just doing it with an extra 2 dimensions isnt novel enough to get my brain interested

#

sadly this has all added up to the result that like

#

i thought itd take me like a year to write the kernel lol

#

its been a year and change and i have basically the scheduler done, object manager, Ps, some other tidbits

#

and thats it

#

i have at least chiselled away at these things and improved them in the interim

twilit smelt
#

and (bad, but functioning) page swapping

#

i will note though

#

they are almost the same number of lines of code

#

lmfao

#

mintia2 Ke is already substantially more sloc than the final version of old mintia

digital pivot
#

how much simpler was the old mintia's scheduler?

twilit smelt
#

mintia2 Ke:

-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Jackal                          34           4693              0          11376
-------------------------------------------------------------------------------
SUM:                            34           4693              0          11376
-------------------------------------------------------------------------------

mintia Ke:

-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Dragonfruit                     20           1779              0           5139
C/C++ Header                     1             29              1             37
-------------------------------------------------------------------------------
SUM:                            21           1808              1           5176
-------------------------------------------------------------------------------
sterile frost
#

quite the difference

#

c/c++ header

#

🤔

twilit smelt
#

and this was old mintia Ke after the same amount of time working on it as i have mintia2:

-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Dragonfruit                     16            876              0           2374
-------------------------------------------------------------------------------
SUM:                            16            876              0           2374
-------------------------------------------------------------------------------
#

so, Ke is nearly 5x larger

twilit smelt
#

it was uniprocessor only and was simple round robin within priorities

twilit smelt
sterile frost
#

yeah i'm aware

#

i've read through some of the code

#

is your scheduler an original idea or is it based on ULE or something

twilit smelt
#

so i guess i shouldnt be too hard on myself because im roughly on the same track i was on with old mintia in terms of overall debugged code output even though the outwardly visible feature set seems so much smaller

twilit smelt
mortal thunder
twilit smelt
#

but the implementation is mine

mortal thunder
#

and my ke is 4992 loc

sterile frost
twilit smelt
#

yeah it is

sterile frost
#

i see, thanks

twilit smelt
#

so its recognizing it as that

sterile frost
#

ah

#

that's not a whole lot of headers tho lol

twilit smelt
#

well yeah thats not the include directory

mortal thunder
#

hey what was that tool that allowed you to see the growth of the locs in your git repo

#

i forget

twilit smelt
#

it wasnt a tool it was a custom script made by a friend

#

he didnt give it to me he just used it on my repo

#

as a test

#

and showed me the graph

mortal thunder
#

no i could swear i had it too once

twilit smelt
#

years ago

twilit smelt
# sterile frost that's not a whole lot of headers tho lol

this is the whole OS (old mintia)

-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Dragonfruit                    316          28407              0          79260
Assembly                        13           2734             84           6822
C/C++ Header                   149           1747            289           4405
make                            75            185              0            297
Jackal                           1             23              0             70
-------------------------------------------------------------------------------
SUM:                           554          33096            373          90854
-------------------------------------------------------------------------------
sterile frost
#

god damn

twilit smelt
#

thats what i threw away in like march 2023

#

the old mintia vmm was 15kloc iirc

acoustic sparrow
#

implement a real feature

twilit smelt
#

mintia2 vmm will probably be more on the order of 20-30kloc

#

if i had to guess

acoustic sparrow
#

and not incremental improvements

twilit smelt
#

even with old mintia i would have long periods of just like staring at the code and fixing random things

#

and i actually credited my habit of doing that for its long term stability

#

and the fact it didnt collapse under its own weight at any point

#

so its probably a good thing

acoustic sparrow
#

yeah but you have to make progress

twilit smelt
#

another thing i dislike is that old mintia felt like a fun toy and mintia2 feels like a job

#

like it actually looks like kernel code now because its pseudo-pascal and not evil rpn vomit

#

somehow this makes a world of difference in how i feel about working on it

#

like negatively

#

aint as fun anymore

#

also old mintia was like fine to stick with what was okay in 1989 like not having smp and whatever

#

but now i want it to run on amd64 one day and actually run well

#

and that breaks the fun larp a bit

#

someone who followed my project since like 2018 stopped paying attention to it and they said "it just got too professional for my tastes tbh" and that felt very upsetting

#

because i kind of agree

#

no but seriously its a problem

#

it has changed a lot and im unsure i still like what it changed into

#

as much as i liked it before

#

i miss dragonfruit and the shitty lua toolchain

#

and the "limnstation" name which was incredibly bad but at least had more character than "xrstation" does

#

oh well its probably just rose tinted goggles and nostalgia

#

too late now in any case

digital pivot
#

i liked limnstation more but almost compeltely forgot about it

twilit smelt
#

i may have gotten too taken in by the corporate aesthetic of like mid 80s DEC

#

and all of the zaniness left my project

#

it is no longer zany

#

what hath the NT hyperfixation wrought upon LIMNstation...

#

even the boot firmware

#

look at the old

image.png
#

vs the new

screenshot.png
#

WHERE DID THE ZANY GO

#

well its still a little zany cuz you can actually still get to the old menu from the new one lol

#

the new firmware chain-loads the old firmware to boot aisix and old mintia

#

you can set a boot argument for aisix that causes its bootloader to drop into a prompt through the old firmware console

#

and from there you can get to the old firmware prompt and if you type autoboot the old boot menu will appear

#

thats pretty zany i guess

night needle
#

did i miss a mintia rewrite or what is this about

twilit smelt
night needle
#

im skimming through the convo

#

are you just revisiting old code?

twilit smelt
#

in early 2023 i decided to do a new lang and rewrite mintia in it after like 3 yrs of working on old mintia

#

thats the only rewrite

#

and only one likely to ever occur

night needle
#

ah

#

i will hold you to that /s

twilit smelt
#

i am having nostalgia about the old mintia

#

and howt he project has changed in the last like 5 years

#

I miss the old mintia, straight from the 'Go mintia
Chop up the soul mintia, set on his goals mintia
I hate the new mintia, the bad mood mintia
The always rude mintia, spaz in the news mintia
I miss the sweet mintia, chop up the beats mintia
I got to say, at that time, I'd like to meet mintia

blissful smelt
craggy spire
#

There were so many changes... But I haven't looked into anything in depth for some time since I've been pretty busy. I just see new things and make some notes

#

The MM calls MiInsertListSentinel which apparently allows multiple list heads to be inserted under the same key

#

It looks like it uses the partition ID as the key for sentinel

twilit smelt
#

Weird

craggy spire
twilit smelt
craggy spire
# twilit smelt Weird

Yes, "it uses the partition ID as the key for sentinel" and that's wrong
Working set expansion list goes into MiInsertListSentinel

twilit smelt
#

alternatively, inheritance is being applied to an inheritor who already released the lock

#

the latter is scarier because it may imply a UAF of the inheritor thread

digital pivot
craggy spire
#

Typically this leads to either BSOD or priority issues

twilit smelt
craggy spire
#

As for these I don't know

craggy spire
#

When I test in a virtual machine, all priorities seem to revert to their default values, even if they were boosted to real time range. For example:

0xffffa8066ad03143 [Type: void *]
5: kd> g
THREAD PRIORITY CHANGE: 0n12 -> 0n16
 # Child-SP          RetAddr               Call Site
00 ffffb80d`3b2bbed0 fffff806`ebd14566     nt!KiUpdateThreadPriority+0x63
01 ffffb80d`3b2bbf40 fffff806`ebdf7956     nt!KiSetPriorityThread+0x3a6
02 ffffb80d`3b2bc030 fffff806`ebdf699d     nt!AutoBoost::KiAbpApplyTargetPriority+0x176
03 ffffb80d`3b2bc0e0 fffff806`ebc13b90     nt!AutoBoost::KiAbpProcessThreadEntries+0x58d
04 ffffb80d`3b2bc170 fffff806`ebc12083     nt!KiAbProcessPreContextSwitch+0x1d0
05 ffffb80d`3b2bc230 fffff806`ebd0dd5b     nt!KiSwapThread+0x53
06 ffffb80d`3b2bc340 fffff806`ebd31159     nt!KiCommitThreadWait+0x1eb
07 ffffb80d`3b2bc3e0 fffff806`ebd2ec03     nt!KeWaitForSingleObject+0x4c9
08 ffffb80d`3b2bc4c0 fffff806`ebd2e722     nt!ExpWaitForFastResource+0xb3
09 ffffb80d`3b2bc570 fffff806`ebd2e377     nt!ExpAcquireFastResourceExclusiveSlow+0x242
0a ffffb80d`3b2bc760 fffff806`82731b90     nt!ExAcquireFastResourceExclusive+0x217
0b ffffb80d`3b2bc7c0 fffff806`82730a9f     win32kbase!<lambda_63b61c2369133a205197eda5bd671ee7>::<lambda_invoker_cdecl>+0xc0
...

THREAD PRIORITY CHANGE: 0n16 -> 0n10
 # Child-SP          RetAddr               Call Site
00 ffffb80d`3c9676c0 fffff806`ebd1fba0     nt!KiSetPriorityThread+0x246
01 ffffb80d`3c9677b0 fffff806`ebd1edba     nt!AutoBoost::KiAbpUnboostThread+0x12c
02 ffffb80d`3c967810 fffff806`827305f5     nt!ExReleaseFastResource+0x39a
...

So, apparently, this is related to NVIDIA software.

twilit smelt
#

which was asymptomatic until now

digital pivot
twilit smelt
digital pivot
#

if they dont bsod this means they do something wrong in their user space component

#

but how can it get that high real time priority

#

afaik user threads are limited to 15

#

actually i can show you something that might be even more funny

image.png
#

windows internals says that the base priority is never lower than the dynamic priority

digital pivot
#

i wonder what is the case when you want the dynamic priority be lower than the base one

hybrid condor
#

@twilit smelt do you know the meaning of the “deferred readiness” thread state in NT?

twilit smelt
#

I think this is done when the thread is readied under spinlocks it can't actually be put on the ready queues under (out of order or something)

#

So they wait until later to do it for real, possibly as a soft irq at DISPATCH_LEVEL or something with the same effect

#

But I don't remember it's been a while so that could be totally wrong

hybrid condor
#

And what are threads in “standby” state?

twilit smelt
#

Those are threads that have been chosen to preempt the running thread on some processor but it hasn't yet actually switched to it

#

It gets put in a per processor "next thread" field and the processor switches to it asap but until then it remains there in standby state

hybrid condor
#

Only one such thread per processor?

twilit smelt
#

Yeah and if a thread is selected that preempts the next thread then the current next thread is put back on the normal ready queue and replaced

#

Is how it worked last I checked

hybrid condor
#

Got it

twilit smelt
#

This is just to eliminate some extra work afaik and possibly also has locking reasons

#

Basically if another processor noticed this thread will preempt your current thread (and decided to enqueue it directly to you) then why should you have to look at your ready queues? It can just directly tell you what to switch to

twilit smelt
#

@warm pine "OpenVMS Galaxy" was an almost identical project to Cellular IRIX it sounds like

#

"galaxy" comes from VMS's 70s codename "star"

#

because it was basically running multiple VMS kernel instances on a single machine (so, multiple stars. ie a galaxy. corny) and they could dynamically partition the hardware between them and so on

#

it was used on big NUMA Alphas

#

it was done under Compaq and HP

#

between like '99 and '03

craggy spire
twilit smelt
craggy spire
craggy spire
#

One of his hypothetical scenarios was reverting back priority boosts in the case of priority inheritance. As an alternative to lowering the priority immediately on release or doing lazy lowering, you can temporarily lower the priority even below the base priority in order to keep the low priority thread's effective priority comparatively consistent over time

twilit smelt
#

do you guys think its sane to have a kernel primitive which atomically pins the current thread to the processor it is currently running on, which is to say, it just prevents it from being able to be migrated, and then returns the ID of that processor

#

and another which unpins the current thread

#

and this can be used to safely access certain per-cpu structures without disabling preemption

#

the overhead is taking the thread spinlock to set thread^.Pinned = TRUE and then taking it again to set thread^.Pinned = FALSE basically

#

the use case i have in mind for this is page frame zeroing

blissful smelt
#

do you have existing thread flags

twilit smelt
#

i have to disable preemption for the duration of using this (i.e. the whole time im zeroing the page frame) and the benefit is that i dont need to do a tlb shootdown, i only need to flush the page on the current core

blissful smelt
#

you can use a byte where each bit is the state of a thread flag if you want to avoid making another byte for the boolean Pinned

twilit smelt
#

i dislike that preemption is disabled for the entire course of page zeroing

#

so instead id like to atomically pin the thread to the current core and each core has like 4-8 of these virtual pages instead of just 1

#

each has its own blocking mutex

#

the thread trylocks each of them and when it gets one it can map it and flush the tlb on the current core only

#

if it trylocks them all and cant get any of them, it increments a per-core counter and does a blocking lock on whatever mutex that lands on

warm pine
twilit smelt
#

the reason i dont use bit flags for most of these things in the thread structure is that the locking can get complicated (fine grained locking)

#

and i dont want to have to use atomics to update them

twilit smelt
# warm pine this might be a wise idea

instinctively it sounds scary to allow preemption while doing this if these virtual pages are a limited resource per-core but then i remembered i have priority inheritance

#

so actually itd be fine

#

priority inheritance was such a good investment

#

now i dont have to fear as much if i make something high-traffic, preemptible

#

the main thing that was bothering me is that the page zeroing thread is idle priority so, basically anything should be able to preempt it asap

#

but how its implemented currently i have preemption disabled for each page frame that is zeroed

#

which is 1000+ cycles

#

yucky

#

i also dont want to make these virtual pages for quick mapping of arbitrary page frames a global resource because for one there will be more cache line contention over them

#

for two ill then need to do a big tlb shootdown sometimes

twilit smelt
warm mural
twilit smelt
#

do you think im some kind of pleb with an hhdm

warm mural
#

wrong reply

#

but whatever

warm mural
#

but dont you have an identity map at least

twilit smelt
#

on 64 bit i could just zero it directly though yes

twilit smelt
warm mural
#

oh that wouldnt work for user threads

twilit smelt
#

and i can use that if the page frame just so happens to lie within the mapped range

#

so like the low 256-512mb of the phys addr space or something

warm mural
#

dont think thats worth it

twilit smelt
#

which only support up to 64mb and 256mb respectively

warm mural
#

oh yeah you're right didnt think about that

#

so in fact the hhdm can be very small

twilit smelt
#

yeah also i dont need to create page tables to map it for regions that dont actually exist

warm pine
#

i don't need to support a full 4gib of ram

warm mural
#

thats really neat because you have tons of free virtual memory

warm pine
#

it is perfectly possible to have, but no actual m68k platform has ever allowed this

twilit smelt
#

im unsure this would be thaaaaat much faster in practice because the time is most likely to be dominated by the page zeroing itself anyway

warm mural
#

yea but you can use it for other stuff as well

twilit smelt
#

and the cost would be yet another thing that consumes a constant percentage of all of phys mem

#

on top of pfn db

twilit smelt
#

old mintia only used arbitrary page frame mappings for two things

#
  1. page zeroing
  2. PTE writeback upon page frame reclamation (to indicate the page frame was no longer in memory) when the page was tracked in a process page table, and process attachment wasn't available because at IPLDPC
#

i still do #1 but i wont do #2

#

a page's location in the pagefile will never be tracked in process page tables like that

#

cuz disposable page tables

mortal thunder
warm mural
#

You have an additional structure to do it

#

Something like an amap

mortal thunder
#

Well each page can go in vastly different places in the page file

twilit smelt
#

will have page arrays

#

page tables wont work like they used to, when the last pte is invalidated in a page table itll immediately be destroyed

#

rather than still containing needed info (like where the pages were put on disk) and just becoming pageable

mortal thunder
#

I see

#

So each VAD will have an array of pages that describe this info?

twilit smelt
#

thats kinda sorta what an amap is yeah

mortal thunder
#

I see

twilit smelt
#

this also makes fork a lot saner to implement

#

cow fork

#

because you dont need to convert private pages that are tracked in page tables

#

into shared pages

#

you can just share the amap structure

#

this doesnt lead to the mach object chaining problem because the uvm guy figured out more cleverer ways to deal with the amaps

#

so its only ever a two-level lookup structure (level 1 is amaps followed by level 2 which is objects, the page fault handler looks them up in that order)

#

rather than an arbitrary N-level structure like in mach

#

where object chains just keep getting deeper as you fork

mortal thunder
twilit smelt
digital pivot
twilit smelt
#

the codepath for that can be a little shorter than that for general affinity setting

mortal thunder
#

I actually don't know why, but the effect is that a thread wakes up on the same core it went to sleep on

#

Ah, no, I remember why this is important. Apparently I had issues when the wait timer (that handles wait-object timeouts) was bugged because I was cancelling the timer from a different core

#

ah, yes, and since I use an RB tree instead of a calendar queue or whatever, I end up needing a pointer to the queue that the timer belongs to to remove it...

twilit smelt
#

i just loop locking whatever queue is specified in the timer and checking once i have the lock that its still in that same queue

#

if not i retry

#

once i catch it in that same queue its guaranteed not to be able to move anywhere

mortal thunder
#

Ah yeah I can store a backpointer within the timer

heady bobcat
#

CPU pinning itself is definitely sane and something I believe Linux has.

#

@twilit smelt I'm kinda curious: what is your view on the MCS spinlock and similar techniques?

warm mural
#

That puts zeroed pages on a queue

#

Afaik

twilit smelt
#

In the new idea I had, there would be multiple such pages per cpu and they'd be guarded by blocking mutex

#

I was describing how it is currently which is a single virtual page per CPU where the user of it blocks preemption while they're using it

#

I disliked that because disabling preemption for the whole course of zeroing a page is bleh

warm mural
#

Isn't zeroing a page pretty fast tho

heady bobcat
twilit smelt
#

Linux reinventing another NT idea

#

Except pushlocks are cooler cuz they're blocking locks

#

And they do what the MCS spinlock does basically

twilit smelt
#

so that's not good

#

It's probably the like second or third worst case longest preemption blocked area in the kernel

#

First by far if everything misses in the cache

heady bobcat
twilit smelt
heady bobcat
#

no

#

because you flush the TLB after mapping the page

twilit smelt
#

True I guess that would work

#

Since no other cpus need to see the mapping you made

#

Cuz migration is disabled

#

Makes sense

heady bobcat
#

I know that your architecture has a number of 'fixed' TLB entries that are never overwritten by the TLB miss handler. If you have one such TLBe to spare, it is possible to get even smarter and use that so you have one virtual page shared by all tasks (that TLBe can be reloaded similarily to your equivalent to cr3 on task switches) and no locking/etc. for temporarily mapping a page.

#

(the CPUs never see each other's mapped pages because they map by writing directly into the fixed TLB entry)

#

the disadvantage of this is of course:

  • permanently consumes one TLB entry
  • not portable
twilit smelt
#

I could make page zeroing be architecture specific

#

The main issue I see with that though is that writing the new TLB entry each time I switch threads could be costly

#

But I only need to do it if the thread actually wants it

heady bobcat
#

do you not need to anyway in the common case, for the page tables?

twilit smelt
#

I don't know that that's always the common case

hybrid condor
twilit smelt
#

they used a time machine to steal the NT idea

#

typical unixoids

hybrid condor
#

Why do you need a per-CPU page zeroing thread

dense vigil
#

Slow imaginary computer

twilit smelt
#

There's just one

#

Soaks up idle time zeroing pages so you don't have to do it when you have work later

dense vigil
#

They used to do it on x86 until it started to have a net negative effect

sterile frost
#

is it smart to do even with nowadays amounts of compute?

twilit smelt
#

No they still do it

#

NT still does it

sterile frost
#

i wanted to do it aswell, it seems like a smart optimization

hybrid condor
twilit smelt
#

We showed the dfbsd thing where the guy is like "idle page zeroing bad now because cache" to our NT friend and he said "our internal studies disagree completely"

dense vigil
#

Well idk but the cost of page zeroing is almost nothing these days and u get it in cache which is a bonus

sterile frost
#

i don't think destroying the caches of hot cpus by zeroing out pages is smart

twilit smelt
sterile frost
#

better do it on a cold cpu

#

during idle time

twilit smelt
dense vigil
#

On Linux they measured it as a net negative

sterile frost
#

only do it on a hot cpu if needed

hybrid condor
#

Hm, I don't think we do that in the background

hybrid condor
twilit smelt
#

They at least THINK it's still good

mortal thunder
#

The other time is when a page is allocated from the free non-zero list

hybrid condor
sterile frost
mortal thunder
#

yes

twilit smelt
#

(non temporal)

craggy spire
twilit smelt
#

also probably you should limit it to only zero out about 500kb/sec maximum so you don't take up a lot of bus bandwidth

twilit smelt
craggy spire
#

I was told that they spent a long time improving it

twilit smelt
#

As for my use case it's trivially just better. Like it makes a huge (positive) difference

craggy spire
#

And shows better results

twilit smelt
#

Probably cuz I'm working with something comparable to the stuff classical NT ran on back in the day

digital pivot
dense vigil
twilit smelt
#

So for me it's a no brainer

craggy spire
dense vigil
#

On a slow computer it definitely outweighs trashing caches

#

Slow being like really slow

twilit smelt
dense vigil
#

Hmm yeah ig

#

Does your arch have that?

twilit smelt
#

Perhaps NT's special sauce is NT. writes.

twilit smelt
#

zero through a noncached mapping

dense vigil
#

Nt store is smart actually idk how I didn't think of that

digital pivot
dense vigil
#

But iirc its an extension on x86

twilit smelt
#

Only issue is I'd have to do a dcache shootdown before releasing the zeroed pages for allocation

dense vigil
#

Why, if its not in the cache?

twilit smelt
#

Because there can be stale dcache data for the pages I zeroes

#

zeroed

#

Idk if NT writes are cache coherent on x86 but they aren't for me

dense vigil
#

Shootdown before zeroing or after?

twilit smelt
#

After

#

To make sure they actually read zeroes if they allocate and use this page

#

And not some random stale dcache junk

#

I could batch it though

dense vigil
#

Ah yeah

#

You should do that probably

twilit smelt
#

Just once every second or so wouldn't be noticeable but also my caches are tiny. If they were big it'd be a much larger tragedy to do this shootdown

sterile frost
#

why shootdown eagerly instead of just flushing the dcache when you pop a page off the empty queue?

twilit smelt
#

So probably on a modern machine it's not worth it if NT writes aren't automatically invalidating of dcache lines and stuff, if they are then it's a good idea

twilit smelt
craggy spire
dense vigil
#

Since u have to wait for all cores to pop it

sterile frost
#

oh smp, right

twilit smelt
#

You don't need to zero as quickly as possible

#

You can do a leisurely few hundred kb per second or so

#

More if you can tell the other cores are also mostly idle

#

If they are then you can go full tilt I guess

dense vigil
#

Sounds like its only beneficial on modern systems if you're extremely careful with it

twilit smelt
dense vigil
#

That's why Linux opted not to do it

twilit smelt
#

Each core can do its own dcache invalidation if the global sequence number doesn't match the one stashed in its prb when it tries to grab a page from the zero list

dense vigil
#

Fair

#

You can only invalidate the entire thing?

twilit smelt
#

Global sequence number is incremented each time a batch of zero pages is released to the list

twilit smelt
dense vigil
#

Linux loves seqnums, they're literally everywhere

dense vigil
twilit smelt
#

Yeah

dense vigil
#

Perhaps

twilit smelt
#

My calc 3 exam is about to start

#

Gonna fail

dense vigil
#

Calc sucks

sterile frost
#

good luck

digital pivot
twilit smelt
#

I'm doing a math minor so I'm stuck with it

dense vigil
#

But yes mostly as a growing id for synchronization

digital pivot
dense vigil
#

Seqlock is one example of seqnums yeah

#

Does minitia use seqlocks?

heady bobcat
digital pivot
#

they said they solved cache issues with non temporal writes

#

or something like that

heady bobcat
#

something which I believe is relatively common is that you have two studies for X and Y really similar things that show completely different results, and without knowing both in great detail it is easy to (mistakenly) think that the studies study the same thing.

#

this effect is amplified because there are many tiny tiny details that can differ, 1% difference here 1% difference there and suddenly you end up with a 20% difference

dense vigil
#

Each console stores the last sequence number it has output

twilit smelt
dense vigil
#

how did the exam go

twilit smelt
#

It went great! I'm a 4.3 gpa student! Recruiters hmu!

#

(Could have been better)

#

Turns out it was a bad idea to take a 1 month version of the calculus 3 class in the middle of rail construction and mental health crisis

dense vigil
#

What does rail construction have to do with this?

#

Lol

twilit smelt
#

Made my round trip to class and back like 4-5 hours instead of 2

#

It was also extremely unpredictable so sometimes I'd think I was leaving with plenty of time and there'd be an extra bus bridge for just specifically that day and I'd get to class 40 min late

dense vigil
#

Holy shit

#

Us distances are crazy

twilit smelt
#

1 month calculus 3 class that covers like 1.5 weeks worth of material every lecture

#

That I was often late to or missed entirely