MINTIA (not vibecoded)

[dense vigil]

oh ok

[twilit smelt]

I just don't want him to waste time on smth I've been planning to scrap for forever

@sterile frost in case there was a misunderstanding on this part lol

[sterile frost]

yeah no i got that

i was just looking for solutions to my problem lol

[dense vigil]

whats the problem?

[twilit smelt]

Also I def would not refer to my compiler as a "proper compiler"

[sterile frost]

but the pipeline is more proper than anything i've ever written

@dense vigil

[dense vigil]

ah

[twilit smelt]

Oh yeah the way I did that isn't rly applicable to anything you're doing huh

[sterile frost]

no unfortunately it's tied to the jackal parser so that's unlucky

[twilit smelt]

"more proper" =/= proper

100 is a better approximation to 0 than 10000 but its still not "correct"

i dont even do SSA man

[sterile frost]

u know what that's totally fair lol

the backend seems pretty well written though, but i'm not surprised after seeing mintia

i think that for my struct offset problem i might emit a custom .h file with C style defines at the end of parsing in jackal

of course it's very specific for my amd64 port but whatever, i don't expect this to be upstreamed anyway it's full of hacks atm so i'll consider this one a hack as well

[warm mural]

Why not make the jackal struct layout the same as C

If that's a thing

[sterile frost]

the layout is not the problem, the problem is that i can't be fucked hardcoding all the offsets and figuring out what offset something should be at

especially since existing jackal code already makes use of this feature, the compiler creates special defines for inline assembly

i just generated a C header with pp defines that i include in my GNU AS sources

[warm mural]

I mean you could

Just put a comment

[sterile frost]

i still need to figure out at what offset a given field is

and in a big structure like KiPrb it's annoying

[twilit smelt]

hows this work going

oh shit i said id do the elf thing

lol

[sterile frost]

yea lmao u did

idk i kinda forgot zbout that, i've been working on my own thing recently

but i did make a thing that can generate a header with jackal struct definitions for use in assembly files

so that will be helpful until proper xrasm support comes

u said u don't want (me) to add a x86 backend because u want to rewrite all of it anyway right

[twilit smelt]

yeah i need to re-do the middle end and backends

before investing in an amd64 backedn

[sterile frost]

yeah makes sense

[twilit smelt]

basically bc id have to redo the amd64 backend later

once i redid the middle end

like theres no reason you couldnt do one now and have it work

it would just SU CK

[sterile frost]

is this also why you do some interesting things around the code base

like extracting struct members to local variables

to get better codegen i presume?

or is that purely your code style and wasn't done with that in mind lol

i cant find an example anymore, my bad forget about that

[twilit smelt]

probably that but if its in concurrent code it could also be to capture the state of some data at a particular moment in time

[twilit smelt]

mintia has a lot of really insane concurrent code

well by hobby kernel standards

the scheduler and the blocking mutexes in particular

[heady bobcat]

insane as in... good code? ...or madness?

[sterile frost]

probably highly concurrent

but it might be misunderstanding

it probably is lol

[twilit smelt]

well u can decide https://github.com/xrarch/mintia2/blob/main/OS/Executive/Ke/KeTurnstile.jkl

p sure ive linked you this file before lol

[heady bobcat]

yeah thanks for reminding me to take a look at that lol, totally forgot.

[digital pivot]

was it you had some interesting nuances with trylocks?

[twilit smelt]

well to put some perspective

a weird ass trylock was the much better solution vs what i was doing previously

[heady bobcat]

this code isn't too bad tbh. it's just really hard to read for me beacuse of the language

[twilit smelt]

its basically C

reskinned

lol

[heady bobcat]

yea, I can see the -> → ^. pointer syntax stuff

but it's like someone from sweden trying to understand danish or norwegian

sure

it's possible

but it takes longer to understand

[sterile frost]

it's really intuitive for me, i like it

[twilit smelt]

thats basically purely for aesthetic reasons

to be specific i was trying to make it look like pseudocode from the NT kernel design notes from like 1989 lol

(which itself was reminiscent of pascal)

[heady bobcat]

that's an... interesting place to get language design inspiration from

[twilit smelt]

PROCEDURE QueueApc (
    IN Acb : POINTER KtApc;
    IN Tcb : POINTER KtThread;
    );

BEGIN

    IF Acb.Mode == Kernel THEN
        IF Acb.Type == Special THEN
            Insert APC at front of thread kernel APC
                queue selected by Acb.ApcIndex;
        ELSE
            Insert APC at end of thread kernel APC queue
                selected by Acb.ApcIndex;
        END IF;
        IF Tcb.State == Running AND
           Acb.ApcIndex == Tcb.ApcIndex THEN
            IF Tcb.NextProcessor == CurrentProcessor THEN
                Set software interrupt at IRQL 1;
            ELSE
                Set APC delivery request for target
                    processor;
                Set interrupt request for target
                    processor;
            END IF;
        ELSEIF (Tcb.State == Waiting AND
              Acb.ApcIndex == Tcb.ApcIndex AND
              Tcb.WaitIrql == 0) AND
              (Acb.Type == Special OR
              (Tcb.MutexCount == 0 AND
              NOT Tcb.KernelApcInProgress)) THEN
                Unwait thread with status of KernelApc;
        END IF;
        Tcb.KernelApcPending = True;
    ELSE
        Insert APC at the end of thread user APC queue
            selected by Acb.ApcIndex;
        IF Tcb.State == Waiting AND
           Acb.ApcIndex == Tcb.ApcIndex AND
           Tcb.WaitMode == User AND
           Tcb.Alertable THEN
            Tcb.UserApcPending = True;
            Unwait thread with status of Alerted;
        END IF;
    END IF;
END QueueApc;

heres an example of some pseudocode from the NT design notes

[twilit smelt]

there is a lot of lore for why lol

[digital pivot]

advanced locking code in hobby projects is rare, its always interesting to see one

[digital pivot]

any plans for more cool stuff?

like locking

[hybrid condor]

And do you need it for a hobby project

[twilit smelt]

the hobby is to learn kernels as deeply as i can

which justifies some forays into absurdly complicated things that nothing im doing actually requires

[hybrid condor]

That guy up there says it's rare in hobby projects : well no surprise there, really

[twilit smelt]

not that this turnstile impl is anywhere near the complexity or usefulness of like the xnu implementation but its probably the fanciest lock in a hobby kernel

afaik

[hybrid condor]

macOS/iOS have turnstiles and GCD, where it's really important since the system has a lot of heterogeneous priorities across processes

[twilit smelt]

gcd?

[white valley]

greatest common divisor

[hybrid condor]

Theres also a lot of IPC, and it supports this as well

[hybrid condor]

Grand Central Dispatch

google it

[twilit smelt]

i have heard of it

i havent seen the GCD abbreviation so i got confuz

[hybrid condor]

XNUs IPC is a treasure trove of cool solutions, especially for synch. A lot of work has been put into IPC

mach ports are placed into a dedicated memory partition instead of being mixed with other allocs
once a memory has been used for a port, it can only ever be used for ports, even after being freed
a port can be i) in use ii) freed but reserved iii) unmapped
by ensuring ports remain stable and not get repurposed the kernel to perform speculative locking optimizations

partitioned alloc + hazard pointers =>lockfree port lookups

very important for lots of things, eg core audio which heavily relies on mach semahpores

[twilit smelt]

we found out a phrase for this which is "type stable memory"

i have no idea the origin anymore @warm pine said the phrase and it stuck in my head

[warm pine]

it's from the CacheKernel

which was briefly subject to attention in the 90s

which helped inspired IBM's k42, and that had some influence on linux later, but oddly the term seems not very popular

[hybrid condor]

there are actually a lot of interesting quirks when it comes to synch

eg when a trylock chain fails the mach ipc often takes a super lock

and the ntries again

You can spend a long time digging into it, though the code is quite convoluted

similar thing https://chromium.googlesource.com/chromium/src/+/master/base/allocator/partition_allocator/PartitionAlloc.md

This is one of those cases where a security improvement sped up some important kernel code

in this case mach ipc

[twilit smelt]

@acoustic sparrow i just switched to using a string structure that tracks the length of the string everywhere in mintia2

one cool thing i didnt realize is that this also speeds up the common case of string compare a lot

you can check the lengths

for equality

before running along the strings

[acoustic sparrow]

ah yeah nice

you can also store a hash

[twilit smelt]

only if the hash is homomorphic with respect to the lexicographical value of the strings

or whatever

[acoustic sparrow]

is that really important though

[twilit smelt]

well i use string compare to walk along avl trees keyed by strings in at least two places

[acoustic sparrow]

do you need to store e.g. directory entries in lexicographical order

or just "a" consistent order

[twilit smelt]

i suppose any consistent order would work

for this purpose

[acoustic sparrow]

because if it's the latter you can compare hashes as the first step

yeah

[twilit smelt]

namely the interned string buckets and the children of name cache entries

[acoustic sparrow]

both of these don't really care about the precise order

[twilit smelt]

EXPORT FN RtlCompareRtlString (
    IN str1 : ^RtlString,
    IN str2 : ^RtlString,
) : UWORD

    len := str1^.Length

    IF len != str2^.Length THEN
        RETURN len - str2^.Length
    END

    i := 0
    data1 := str1^.Data
    data2 := str2^.Data

    WHILE i < len DO
        IF data1[i] != data2[i] THEN
            RETURN data1[i] - data2[i]
        END

        i += 1
    END

    RETURN 0
END

[acoustic sparrow]

this is not lexicographical already

so yeah a hash would totally work

[twilit smelt]

how

[acoustic sparrow]

lexicographical would put "a" before "ab" which comes before "b"

and RtlCompareRtlString would give you "a", "b", "ab"

[twilit smelt]

true.

[acoustic sparrow]

and ofc hashes are faster to compare

oh also, one more possible optimization you can do is if data1 == data2 then its always equal

[twilit smelt]

thats probably not common enough to be warranted

[acoustic sparrow]

yeah probably

[twilit smelt]

also knowing the length ahead of time makes it easier to do like bulk comparisons of multiple characters at once

so thats another one

[acoustic sparrow]

yeah

but if you compare hashes first it doesn't really matter

because the slow loop will almost always be a full match

[twilit smelt]

computing a hash every time i initialize a string is gay

[acoustic sparrow]

its pretty fast, though

but yeah i guess

you could lazy compute the hashes, maybe?

[twilit smelt]

updated:

OsSetGroupProcess
OsSetBasePriorityProcess
OsExitProcess
OsQueryProcessCount
OsQueryProcesses
OsSetQuotaProcess
OsQueryQuotaProcess
OsOpenGroupByPid
OsSignalGroup
OsSetSessionProcess

[twilit smelt]

OsSetGroupProcess will be basically setpgid except takes handles as arguments instead of pids

process handle and process group handle

OsOpenGroupByPid opens a process group by pid (duh)

yes i am implementing posix session and group semantics directly in my Ps. mad?

OsSetSessionProcess is setsid except it takes a process handle argument

i am implementing these in order to get tty signal routing and stuff right

and to make a bash port and stuff relatively painless under the linux subsystem

job objects will still be the inescapable session container

[twilit smelt]

ive been thinking about it for a few weeks and came up with a way to make it work nicely

[twilit smelt]

this is one place where the posix api is. somehow. kind of well-designed

the exact restrictions i was hoping for in order to make my desired implementation possible, are in there

someone must have been thinking alike at one point

[twilit smelt]

CUT MY LIFE INTO PIECES

THIS IS MY LAST RESORT

[dense vigil]

lmfao

[dense vigil]

did you see this https://www.reddit.com/r/osdev/comments/1i01b73/my_first_os_with_a_gui_im_so_happy/

this is just a shitty kernel-level terminal emulator and it got 5k

[hybrid condor]

yeh this project is clearly dying

[night needle]

r/osdev is a cesspool

[twilit smelt]

yes...

[twilit smelt]

@acoustic sparrow challenge: find a major vuln in old mintia

[acoustic sparrow]

nahh that sounds hard

and requires doing more than using boring trivia

[twilit smelt]

it's probably not hard

It's prob like intermediate

[acoustic sparrow]

im just lazy tbh

lol

would be much better if new mintia had a userspace to launch attacks from tho

[twilit smelt]

There's like 10k+ lines of completely untested code in mintia2

[acoustic sparrow]

but it has no userspace

and so i cant really try making a challenge out of it

[twilit smelt]

Yea

I'm just saying it'll be an easy target for the first few months it has a userspace

Lol

[acoustic sparrow]

😦

[twilit smelt]

I should be writing unit tests but that's such ass

to do for kernel components

I'll be writing tests in userspace like with old mintia and I'll make sure those have good coverage

[acoustic sparrow]

id prefer if you got to userspace in like. a week or two

[twilit smelt]

Unfortunately it'll be like 6 months at this rate

[heady bobcat]

I have unit tests for two things: mutexes and vmatree (an augmented AVL tree)

[twilit smelt]

But you know I'll get there I have a proven record of saying "it'll be done in 6 months" and actually doing it (in 12 months)

[heady bobcat]

other than that, I just see if it

[acoustic sparrow]

because thats about when i have to get any chall i want to get into kalmarctf ready :^)

[heady bobcat]

wait kalmarctf?

[acoustic sparrow]

yes?

[heady bobcat]

are you in kalmarunionen?

[acoustic sparrow]

yes

[heady bobcat]

nice

actually I think I've seen you in the midnight sun CTF discord lol

[acoustic sparrow]

quite possibly because i was playing lol

[twilit smelt]

still be in it whenever the next one is so that u can get mintia in then

[acoustic sparrow]

assuming they dont kick me out for a distinct lack of being danish

[twilit smelt]

Is being danish a rule

[heady bobcat]

the way I understand it it's a guideline

[acoustic sparrow]

no but they are all danish

lol

(i think)

or most of them are idk tbh

[heady bobcat]

is kalmarctf open or are there quals or something?

[acoustic sparrow]

open

[heady bobcat]

nice

[acoustic sparrow]

you should play :^)

[heady bobcat]

I'll 99% participate

though I will have to setup something better than su <low-privileged user> for sandboxing

[acoustic sparrow]

nah dont worry i promise i wont put that much malware in at least my challs

[heady bobcat]

oh fk off... it is 7th to 9th march which collides with the finals of a national ctf I am in

womp womp

[acoustic sparrow]

:((((

fullsolve the national ctf and then play our challs then

[heady bobcat]

haha

or just bribe you for the flags á la cursedctf

[twilit smelt]

I'd suck at competitive hacking i think

I haven't bothered trying it at least

[heady bobcat]

definitely do

it's fun

[twilit smelt]

Nah

[acoustic sparrow]

you should do it!

[twilit smelt]

Breaking stuff isn't my thing unless it's my own stuff

[acoustic sparrow]

try it

march 7th to 9th, we have at least one thing that can be classified as an 0day planned

[heady bobcat]

0days are fun

[twilit smelt]

last time i did anything nefarious with a computer was when i was like 10 i figured out i could port scan mc server hosts to find ppl's badly secured servers to grief with my sister, i even got into the control panel for some of them

Apparently this later became a really popular skid thing to do but i swear i came up with it on my own

[acoustic sparrow]

lol

[heady bobcat]

ok someone linked me this commit:

[acoustic sparrow]

what is that 1

[heady bobcat]

the purpose of ctfs is for people to be nefarious against the challs

[acoustic sparrow]

lol

[twilit smelt]

Ik

[heady bobcat]

the bestest, most secure RSA exponent

[acoustic sparrow]

oh! yeah

i love a good exponent

[heady bobcat]

additionally it offers better performance

since d=1

[acoustic sparrow]

yeah

high security lmfao

[heady bobcat]

no complicated totient function or modular multiplicative inverse needed

[acoustic sparrow]

genius

[heady bobcat]

[acoustic sparrow]

okay im gonna work on my os now

[white valley]

hmm

that actually happened to me once

some assholes set up bots that would scan ip addresses and automatically join and grief the servers

i had whitelist etc. on but they spoofed my friend's username and got in anyways

had to disable some setting where you can see the list of players in a server without joining it

[twilit smelt]

Think you were just griefed by your friend

[heady bobcat]

you would either have to be running in offline mode (unlikely), or the griefers would've needed a zeroday against minecraft (unlikely), or your friends account was pwned (somewhat more likely) or your friend did the griefing

[white valley]

it was offline mode

so it is a likely scenario because my friend doesnt have minecraft

told him to buy it

refuses

[white valley]

no they started spamming bullshit in chat

advertising an actual griefing thing

[heady bobcat]

like people are going to be like: "ohno I got griefed! as revenge I will grief even more people"

[white valley]

it didnt even matter i had backups lol

[heady bobcat]

did your friend buy the game afterwards or what?

[white valley]

no

also it would be really damn funny if he was the griefer cuz he literally woke me up in the middle of the night with a phone call telling me the server was being griefed

[warm pine]

exactly what a griefer would do

[heady bobcat]

don't make him question his reality by telling him his friend griefed him

[warm pine]

anyway it's exactly as a griefer would do

[twilit smelt]

Your friend griefed you this sounds exactly like something I'd do at age 13

[white valley]

bro he had hundreds of hours of work in that server

also i checked the server logs

unless he literally git cloned the griefing bot from these guys and ran it on my server to troll me theres no way

also he didnt know whether there were any backups or not so i dont think he would risk it

[heady bobcat]

theory: he wanted to quit playing and was secretly hoping that there weren't backups

[white valley]

lol

that would be funnier because i open the server on demand

[white valley]

the bot logged in the moment his internet cut out

i know this because it was almost exactly 30 mins after he last disconnected

his internet cuts out every 30 mins

[heady bobcat]

no... he was faking it all along

[white valley]

if he was faking it, it was the most annoying troll of all time

[hybrid condor]

You really should, as it's important

though it's not quite enough for a perfect experience

In any case it'll definitely make the project stand out from the hobby crowd

[twilit smelt]

OsSetGroupProcess
OsSetBasePriorityProcess
OsExitProcess
OsQueryProcessCount
OsQueryProcesses
OsSetQuotaProcess
OsQueryQuotaProcess
OsOpenGroupByPid
OsSignalGroup
OsSetSessionProcess

I AM GETTING EVERY SINGLE ONE OF THESE DONE TODAAAAAAY

[acoustic sparrow]

do it

is there any reason you cant do userspace?

[twilit smelt]

there is no io system or vmm

at all

[acoustic sparrow]

ahh

okay yeah

[twilit smelt]

old mintia took 12 months to get to userspace bc i refused to do anything in usermode until i was demand paging the executable off a filesystem

[acoustic sparrow]

thats a ton of work

i see

[twilit smelt]

which i had no idea how to do mind you

i coudl have gotten to that same point in 6 months this time if i didnt get one-two'd by a motivation slump and then starting first semester at proper uni

but unfortunatley its been 6 months and i have like a scheduler and some physical memory management and process stuff and some stubs for namespace stuff and thats about it

thats further than i was in 6 months with old mintia tho i think

[twilit smelt]

@acoustic sparrow i have something to say to u

u are going to go further than me more quickly despite the fact i am both older and have a like 5 yr head start

the reason for this is that you are less annoying than i am

so remember that

[acoustic sparrow]

well yeah idk

you have made an operating system that is a lot more impressive than anything ive made

because i cant concentrate on literally anything lol

[twilit smelt]

this means essentially nothing in terms of anything concretely related to success

being non-annoying is much more valuable for that

[acoustic sparrow]

yeah maybe

[twilit smelt]

i get blocked by cool ppl because i was too smug about knowing 1% more about a niche topic than they did

you will get a $300k/yr job offer from one

[twilit smelt]

part of the motivation slump ive been having is that mintia looks like a real kernel now that its written in a non horrifying language

and i think i have a psychological barrier where a Real Kernel sounds a lot scarier or more annoying to work on

like

the "cute toy" vibes of limnstation are gone

[heady bobcat]

a Real Kernel™️ would be kinda cool to have though

[twilit smelt]

atp my project is to like recreate the software infrastructure of a fictional risc workstation company from like 1989-1991

this sounds fun until you start trying to do it really well and then suddenly youre basically working for free for a fictional risc workstation company from 1989-1991

thats a lot less fun

im working for free as a 100x programmer for my fake boss at XR/corp backwards in time in an alternate reality

on a purely vibes basis i kind of unironically believe in alternate timelines and i think it might be out there somewhere.

theyd probably think im a fucking weirdo

[heady bobcat]

speaking of vibes... XR/corp has a dystopian vibe to it...

I find it easy to imagine like the ruins of a lifeless city with mega scyscrapers everywhere with XR/corp logos plastered all over the place

[twilit smelt]

theyd never be that successful

theyd enjoy moderate success in some niche like academia and scientific computing until like the mid '90s but then theyd suddenly get kneecapped by market forces and get bought out by the mid '00s

also i dont have like really robust lore drawn out

it might not be called xr/corp at all it might be XRSI or something

(Xyzzy RISC Systems Inc)

[warm mural]

ibm bought xr corp and it fell off

[twilit smelt]

(and ofc thats what XR stands for)

[heady bobcat]

eXtremely Reasonable Corporation

[twilit smelt]

i hope however annoying i am that ppl dont associate that with my project

my project doesnt embody me at all it embodies a bunch of fictional bearded systems guys from the 90s

if it embodied me itd be called like yaoiOS or something

so thank god it doesnt

[white valley]

mintia is programmed entirely by unpaid interns

both in lore and irl

[twilit smelt]

Guh

I forgot to push my work before heading to class now I can't work on it instead of paying attention to the calc 2 lecture

oh well it wasnt much

ill just write it again

[white valley]

who uses differential equations anyways

buncha nerds

[twilit smelt]

me bc im hoping to graduate with a math minor

[white valley]

are you gonna be a quant too

[twilit smelt]

with a math minor from a state school? not likely

[white valley]

if we're being serious, what jobs does a math minor get you

or a math major even

outside of academia

[twilit smelt]

Math minor just makes ur cs degree look less fake

Lol

[white valley]

fair

though if you have mintia on your resume i dont think anyone's gonna question the validity of ur cs degree

[twilit smelt]

Mintia is not impressive

No tests and no docs, only runs on a fake computer

[glossy leaf]

even then, compared to a lot of other things it is

[white valley]

i think the fact that you're doing it solo might be more of a deterrent idk

i would at least try to accept / do contributions to look more like i can work with a team

also the fact that you managed to write that thing in dragonshit would probably tell whoever's hiring you that you can handle their horrid codebases with no problems

[twilit smelt]

It will be impressive when it has tests and docs and runs on a real computer

[white valley]

just because it hasnt met the end goal yet doesnt mean it's not impressive

[acoustic sparrow]

mintia2 cannot be called a working os

given its major subsystem deficency

[white valley]

it's a non-operating system

[digital pivot]

do companies care about that in the us ?

[twilit smelt]

They might

Depends who looks at it

[digital pivot]

hey i remember you had some docs, a book even

and your research into nt is impressive

which is reflected in mintia

[twilit smelt]

Docs for the architecture yeah

OS is entirely undocumented

[twilit smelt]

It could be a pretty badass DOS.

other than that no. It not work.

[acoustic sparrow]

do you have a disk driver?

[twilit smelt]

No. But that would be part of the minimal effort to get a dumb dos

[twilit smelt]

I'll be writing a book for the OS tho

[acoustic sparrow]

im waiting for Minta 2 Enterprise Edition

with customer support

[digital pivot]

sounds cool

[twilit smelt]

the intention is that itll be an os that u could put on a university mainframe in like 1991 and u could sandbox the students into their own memory partition with their own pagefile on their own paging disk etc and nothing they do can possibly crash or d-o-s the system

and then u can stripe the disk for the home directories across like 8 disks and also mirror it to another set of 8 disks

[acoustic sparrow]

problem

[twilit smelt]

and do all kinds of fun stuff

[acoustic sparrow]

some of those students may be a bit too much into infosec

and also some of them can just shoulder surf the admin lol

[twilit smelt]

cant do anything to prevent human error from trusted operators

thats kind of a given

[twilit smelt]

this is the kind of configuration youd have on some big DEC Alpha server or something

im planning on having disk striping and mirroring and stuff and youll be able to stack these into arbitrary configurations

youll need a regular partition on one of the disks to be the boot partition containing the kernel and boot drivers though cuz the bootloader wont be aware of it

idk where to put the config info though

maybe on that same boot partition and its your job to back it up lol

[twilit smelt]

it would be very fun for ppl to try to break mintia

cuz then ill have stuff to fix

its the only hobby os afaik making a sincere attempt to be difficult to dos and stuff

on all the others im sure there are trivial ways to get the kernel to allocate like all of physical memory and then you die and nobody can stop you

stuff like that

mintia2 will let u sandbox ppl into their own memory partitions even so even if u filled up the entirety of memory and then thrashed the pagefile, someone in another memory partition will be getting by just fine and barely notice

cuz they have their own pagefile on another disk or something

and their own page lists and their own modified page writer etc

and u cant kill system threads by engineering a priority inversion situation cuz theres full priority inheritance

which is blind to see past shared locks, but locks that can be taken by important system threads are never taken shared

even if u defeat priority inheritance and have one or more middle priority threads taking up a ton of cpu starving the lower priority lock holder, now theyre gonna get downgraded to timeshared threads (see ULE) and the thread holding the lock will eventually be scheduled in even if its lower priority

u could create so many threads that they take up tons of cpu time but split up such that they never rly get individually downgraded from interactive, but this would require creating many more threads than your quota allows you to

so it should be p difficult to dos

(if configured correctly)

(nobody but me will be qualified to configure a mintia installation correctly unfortunately)

[twilit smelt]

i lied

ive been guilted into writing unit tests for everything first

@acoustic sparrow how should i do my kernel unit tests this sounds like something youd know a weird amount about

[acoustic sparrow]

i do not

[twilit smelt]

should i just have a build option like BLD_TEST that causes unit tests to be compiled in and called in sequence and messages indicating success are displayed on the console

idk how else to do it really

[acoustic sparrow]

what kind of test do you want, really

[twilit smelt]

unit

[acoustic sparrow]

for a lot of things, you can just. test them on the host

with a fake hal

[acoustic sparrow]

otherwise yeah i think that would work

[twilit smelt]

u could automate it still

after doing all the tests u can write some gunk over the serial port and then write some magic value to a fake emulator-only io port that quits it

a test driver can interpret the gunk to determine successness

[acoustic sparrow]

no need to have a fake emulator-only port tbh

just add a normal shutdown on xr17032

[twilit smelt]

that would be very weird to have in like 89-91

software shutdown

[acoustic sparrow]

in an enterprise type os?

[twilit smelt]

the earliest things i know that really had robust software shutdown were like sparcstations which got it substantially before PCs did but that was in like '93

[acoustic sparrow]

even mainframe type stuff didnt have it?

damn

[twilit smelt]

my '93 sparcstation 20 can do software shutdown

and even software power-up with a key on the keyboard

PCs were going all "IT IS NOW SAFE TO SHUT DOWN YOUR COMPUTER" until like idk

98?

[twilit smelt]

thats a different story

if youre already spending $5 million to build each machine its not rly any pain to add some circuitry like that to support remote management

[acoustic sparrow]

just change up your larp

[twilit smelt]

but the equation is different for relatively cheap machines like workstations

there is no xr/frame

there is xr/station and xr/mp which are teh same basic platform architecture but no xr/frame which would be a way different one probably

[acoustic sparrow]

say that managment wants you to build it in

for future support of a theoretical system

and stupid yes but idk managment has silly ideas sometimes

[digital pivot]

does any other hobby system have unit tests

[twilit smelt]

ive never looked

so i have no idea

when i look at other hobby kernels i zoom straight to the scheduler and the page fault handler

and those two things basically tell me everything about the rest of it lol

but i havent looked for unit tests!

maybe managarm does

i doubt serenity does

minoca might

[acoustic sparrow]

managarm i dont think so

[twilit smelt]

those are where id look

[acoustic sparrow]

serenity lolnope afaik

managarm crashes all the time too

you dont need unit tests, you need an io subsystem

[twilit smelt]

ill make a list of things that need unit tests. generally ill exclude things that are already very well covered by some other stress tests that would cause it to immediately break if anything was wrong iwth it (locks, the background action of the scheduler (not to be confused with explicit fiddling like changing priorities which do need tests), etc)

but other stuff needs unit tests

like the like 10k lines ive written in Ps and not tested literally at all

i dont think its 10k its some large number though

several thousand

[teal trench]

managarm doesn't (currently at least) have any unit tests

we have some end to end (i think that's the right term?) tests

[twilit smelt]

before io system i need to finish process management

[acoustic sparrow]

true

but you dont need a tested one do you

[teal trench]

just use it for long enough to be convinced there are no bugs and then be surprised when it does break

like everyone else

[twilit smelt]

io system is scaring me a little bc im less sure i know how i want to do it after seeing the cool multi plane registry type stuff

[acoustic sparrow]

ah

lol

[twilit smelt]

i want loadable/unloadable drivers and i dont want to do anything to make hotplug really difficult

[acoustic sparrow]

macos has some plist goop for some of hotplug afaik

[twilit smelt]

what do plists have to do with hotplug

[acoustic sparrow]

not just hotplug

basically all discovery/detection stuff

sorry i cant think atm its a bit too late

[twilit smelt]

in old mintia hotplug would have been very difficult to accomplish

it was not considered essentially at all

i imagine the basic principle though would be that if a device disappears while its in-memory structures are still referenced by things, youd cancel all the in-progress IO requests with the appropriate status, and then reject any further ones

and youd just keep the in-memory structures around instead of aggressively trying to clean them up

and theyd go away when refcounts reach 0 (probably as processes using the device (or files in filesystems mounted upon it or whatever) receive unexpected error statuses they dont know what to do with and crash lol)

this sounds like an ugly way to deal with hot-unplug but any other way to deal with it sounds so much uglier

[heady bobcat]

often it's the least ugly thing rather than the bestest thing that we have to do

[twilit smelt]

One thing this results in is like

Say an external hard drive is unplugged and plugged back in

And re mounted

You might have a broken ghost mount that just returns IO ERROR!!! for all accesses, and this exists in parallel with the new working mount

lol

but ofc no new references to the old one can be created

[heady bobcat]

I thought you were going to say something like 'the old VFS structures are still in memory, let's reuse it!'

but that would just be ridiculously stupid

[twilit smelt]

no you'd probably detach it from its mount point and make it impossible to navigate there from outside

You could still do relative lookups from inside if you already had a fd open to a directory

But if that lookup caused any IO (ie doesn't just hit in the page cache) you'd get an IO error return

This entire detached subtree goes away as soon as nobody holds any references to it

[heady bobcat]

actually. wouldn't it make the most sense to keep the mountpoint etc around but any attempt to step into it returns EIO?

userspace can then unmount it

[twilit smelt]

It'd be nice to be able to re mount it in the same place even if bad guys still have references to it

[heady bobcat]

is it not possible to unmount regardless of bad guys still having references to it?

"lazy unmount"

[twilit smelt]

It would be quite difficult

Ah lazily unmounting yeah u could make it explicit to remove it from the hierarchy

[heady bobcat]

yeah exactly

[twilit smelt]

But I don't see any use of being able to look into the ghost mount

[heady bobcat]

if you mount a filesystem over some directory, and that filesystem goes away. should the contents of that directory be exposed instead?

[twilit smelt]

That's what'd happen

[heady bobcat]

yeah. but does it make sense?

userspace doesn't expect a filesystem to just go away. instead of succeeding with lookups into whatever the contents of that directory might be, would it not be better to give userspace an error?

[twilit smelt]

Filesystems just go away all the time

[heady bobcat]

(obviously userspace can and probably will decide to unmount it anyways, but that's a userspace decision)

[heady bobcat]

and wouldn't userspace want to know about that? my philosophy is that it is better to fail hard rather than silently try to continue and potentially get bogus results.

ladies and gentlemen. we broke will

[twilit smelt]

No i was buying donuts

[heady bobcat]

valid

[twilit smelt]

Generally speaking in real life you don't cover up a directory that actually has contents

with a mount point

A removable volume is likely mounted on its own directory

It's very own

[heady bobcat]

no. but maybe a filesystem contains a bunch of records and a userspace program iterates over those records. if the underlying filesystem dies, would it not be better to fail-hard than for the userspace program to falsely believe that the directory is empty?

I know, weird usecase I just made up in like 5 seconds. But userspace is stupid, and the kernel's job is to cope with that.

[twilit smelt]

the program is badly designed

or its user error for yanking out a drive thats in use

or both

also generally you like

delete

the directory

once the removable drive goes away

you dont just leave it sitting empty

so the path to it will become invalid

[heady bobcat]

who deletes the directory? kernel or user?

[twilit smelt]

probably a volume management daemon

that listens for removable drives to be attached, creates a directory and mounts it

and does the reverse too

so usermode technically

but the OS really

[heady bobcat]

so why split the work of coping with a dead filesystem between kernel and the filesystem daemon? doing only some parts of the work in kernelspace (unmounting) effectively constrains userspace into specific behaviors dictated by the kernel

and also. atomicity

there is the window of time between the unmount and the volume daemon removing the directory

[twilit smelt]

so

ur just listing things that might be sources of hypothetical bugs i wouldnt ever write

[heady bobcat]

I'm slowly but surely climbing mount annoying

[heady bobcat]

just read the backlog and saw this. this is basically exactly what I do lol

so I have CONFIG_KTESTS and the "kernel main" just does this*:

#if CONFIG_KTESTS
        run_ktests ();
#endif

*actually it unconditionally invokes run_ktests, but if !CONFIG_KTESTS, run_ktests is defined to be a static inline empty function

[warm mural]

I dont do kernel tests

I do them on the host

[heady bobcat]

that would kinda work for my datastructure unit tests tbh

(which I never run anyways, because my 20kLoC vmatree_testcases.inl takes approximately 42 seconds to compile)

[warm mural]

i have like 1200 tests

for everything

mostly data structures

[digital pivot]

what i don't really understand , what this registry and dictionaries help with?

other than user browsing

[green pelican]

test as much as you can on the host system imo

for kernel tests have separate kernel images and communicate between a host test runner and the kernel test via serial/interconnect of your choice

i've been using canbus over serial to communicate between tests in vms and a host which has been pretty reliable if not a bit clunky

[hybrid condor]

A boot argument (in debug build) that runs tests integrated into the kernel early, before the userland initializes

In case of a panic you can have special handlers for test mode boot

It's fairly simple and works for xnu

xnu also has user mode tests to check kernel interfaces, but this can be added much later on

[twilit smelt]

i dont know.

if i knew i wouldnt be scared.

[hybrid condor]

In terms of significance, the catalogue matters more than this registry

so dont overthink it

[acoustic sparrow]

i suspect matching

both matching a device to its parents and matching a device by userspace apps

[hybrid condor]

matching works through the catalogue

[acoustic sparrow]

ah

fully?

[hybrid condor]

several types of matching in iokit, but it relies on the catalogue rather than the registry

[acoustic sparrow]

i see

what is the purpose of the registry then?

[hybrid condor]

drivers and user mode apps (nowadays in a very limited way) can query the registry

[acoustic sparrow]

ah i see

[hybrid condor]

ioreg(8)

[acoustic sparrow]

i saw that but idk what its really for

[hybrid condor]

you can see live changes in xcode

[acoustic sparrow]

i see

but thats not really what its for is it

like its not there so that you can inspect it in xcode

right?

[hybrid condor]

yeah lol

[acoustic sparrow]

holy shit

[hybrid condor]

basically dynamic in memory db that represents all the live IO objects and is made accessible from user space

[digital pivot]

?

[acoustic sparrow]

its a lot of effort for something thats this useless

[digital pivot]

my understanding is that it's similar to windows registry which also is a lot of effort

drivers and user apps also can query registry but that's it mostly

[acoustic sparrow]

yeah maybe

[twilit smelt]

@hybrid condor (cont from obos channel) most of the stress tests i was doing would have crashed any other hobby OS essentially "by design" even if the stressed codepaths were technically bug-free (bc of panic on oom, total lack of resource limits, and other stuff)

i want to note that like

difficult things like robustness are what im focusing on with my personal work

[signal zinc]

mrrp mrew 🥺 this seems very cool

so how do you handle OOM? fork bombs? if you care to elaborate

[hybrid condor]

It's super important that you're doing this, the more you can cover your kernel, the better

the words of randoms in discord aren't that important lol, but it's important not to slack off in what you're doing

[twilit smelt]

a fork bomb was one of the stress tests i was doing before, basically to test that:

1. it does not crash the system (lol)
2. no matter what, the system remains in a state where you can still totally terminate the fork bomb

this would cover huge swathes of the vmm and error codepaths all over the kernel so i liked running this one

[signal zinc]

:D

[twilit smelt]

multiple instances of it running overnight and stuff

for instance i had page swapping and this would stress that pretty dramatically

[craggy spire]

How's it going?

[signal zinc]

fork bombs always make me think of the Movable Feast Machine, which had agents that stopped fork bombs via themselves fork bombing until the original fork bomb was sufficiently starved

[twilit smelt]

for OOM handling basically a thread just goes to sleep if it needs a page frame and none are available. it wakes up when one is available. this is pretty ubiquitous among big kernels but the question is how you guarantee that one will always eventually be available once a thread is waiting here

if youre linux you have an OOM killer where you just obliterate people to free up pages and this is the last resort once swap is filled up

i went with the NT way where u basically track memory commit, which is how much of ram+pagefile has been "promised" for some purpose (but not necessarily dished out yet)

when u do a new virtual allocation (like mapping some new private memory into ur address space), this incurs a charge against memory commit; if it would exceed the current commit limit then you try to extend the pagefile to raise the limit

if you fail to extend the pagefile then the allocation is failed with an error status

note that demand paging is still done; there is no immediate allocation of memory here

[sterile frost]

how do you extend a page file, isn't it's size set to something ahead of time?

or does "extend" mean something else in this context

[twilit smelt]

by making the page file extensible

its just a file in the filesystem, you can grow it

you could also set it to be a partition or a whole disk and then obviously it cant be extended

[sterile frost]

hm i guess so, so that's what "system managed size" means for windows page files?

[twilit smelt]

probably

[sterile frost]

im assuming so, since it can either be off, sized statically or "system managed" lol

im used to linux swap being either a partition or a file, which i dont think linux will try and make it bigger if it runs out of space

[twilit smelt]

this creates a guarantee that theres always enough room in ram+pagefile to satisfy a virtual allocation if it were to be hypothetically completely faulted upon; there is no longer a situation where a thread can block for free pages in the page fault handler, and none are forthcoming because the swapper cant find any room on disk to dump pages to (swap is full)

the existence of that situation is what necessitates the OOM killer in linux

instead, this moves all the failure cases to a time where it can be an error status return that can be handled (or cause the program to puke, but the point is that a well-written program has the tools to deal with this situation instead of being potentially randomly killed)

[craggy spire]

When this option is selected, the system also takes into account various additional factors such as usage history

[sterile frost]

yeah i was wondering how the system makes sure it doesn't use too much disk space

[twilit smelt]

if u mean mintia2 its going verrrry sloooooowly

[craggy spire]

Yeah, a lot of studying?

or just a crisis

[twilit smelt]

yes but i was in a motivation slump even before uni started up this semester

so thats not a complete excuse

[digital pivot]

similar on macos?

ah iirc they have a dialog box for that

idk if it can kill processes without a dialog

[twilit smelt]

yeah this is my favorite macos UX advantage over linux

ppl in linuxland have wanted to implement something similar but the kernel and window systems arent developed under the same roof so theres no like established way for this to occur

and every time someone tries to come up with a way somebody doesnt like it and it gets stonewalled

[signal zinc]

what if the system used the entire unused disk space as swap 🥺

[twilit smelt]

this is possible if u dont put a limit on the pagefile extension

memory compression is a cooler first line of defense though before having to actually do disk IO

[digital pivot]

mintia 2 with memory compression?

do you have any ideas how it can be implemented in your memory management scheme

[twilit smelt]

i was thinking, compressing all the anonymous pages on both the standby and modified list

and compressing file pages on the standby list (but not ones on the modified list)

and emitting the compressed data into anonymous pages that are mapped into system space but reside in a special working set

those anonymous pages are the only ones that actually go to disk

they get their own modified page list that is destined for the pagefile

[fair jetty]

#introduce-yourself message

No way

[twilit smelt]

OMG IM A CELEBRITY!

no i just know them from another discord

[signal zinc]

:3

mintia has the distinguished honor of being "oh yeah that one next to fox32 on my shelf"

[twilit smelt]

i got no fans

[signal zinc]

how am i supposed to use an OS that doesn't follow classic mac philosophy

[twilit smelt]

He tells me to work on mintia

[glass sparrow]

didnt know cars spoke english

[heady bobcat]

ahh yes. the CEO of XR/corp.

[green pelican]

Its a little known fact that cats can write C++

[dark shard]

Periodically

[twilit smelt]

i fundamentally dont understand the people who want AI to do all their thinking for them

they want this to vanish from the human experience

they must not understand this at all

yet they larp as engineers and smarty types

ai grifters are unironically a huge threat to the human race

well mostly their giga-grifter overlords (elon musk, sam altman) are

[jolly lion]

people are losing their reading comprehension because of phones and tiktok
with AI this trend will only accelerate
and with the inability to understand anything but GPT generated text will come great problems

[tiny swallow]

big if true

[mortal thunder]

i apologise for replying to something from yesterday

but yeah

thats whats happening

its happening right now in your page file! (well, if you run a memory heavy program)

my pagefile rn is 7.8GB in size but it grows as needed

[sterile frost]

thats cool

also np at least its not a week old message lol

[twilit smelt]

reading "a quarter century of unix"

(1994)

sudden dave cutler cameo

hes everywhere

also an incredibly interesting alternate perspective on the death of prism

as far as i can tell this person is incredibly delulu and their perspective is quite narrow-sighted

this is the almost exact opposite of what ive seen from every other person involved in the cancellation of prism and the death of DEC

they also dont seem to actually know why their project (the mips-based decstations) got the go-ahead and prism was cancelled

theyre kind of implying it was because their proposal was technically far superior

no

it was weird bullshit politics

and bob supnik

im sure they didnt get this insight at the time though

maybe they did later

[signal zinc]

🥺 _ _

from the way the first screenshot passage reads, at first it sounded like cutler was having computer problems, called these two guys in as tech support, and them after seeing the problems said "your computer doesn't work" and left lmao

[twilit smelt]

yeah i have no idea waht the point of that story even is

it really just reads like that

LOL

[twilit smelt]

saying DEC died because they paid too much attention to Alpha and not enough to MIPS is just straight up

i mean

i understand their perspective

the decstation was their project and they seem under the impression it was greenlit bc of technical superiority and they have a right to be salty it faded out

but thats just wrong

[signal zinc]

lmao now that i actually understand what is being said, i think it’s just to say that cutler was not pleasant

i presume now that he was proposing some big changes/extensions to the unix system which were incompatible with the project

[twilit smelt]

it sounds like he was trying a unix compatibility layer called "Eunice" (developed at "SRI" and here colloquially referred to as that)

which ran on VAX/VMS

and was notoriously flaky and incompatible

and he had problems (as everyone who ran Eunice did) so he called in the unix experts in the company (guys from the ULTRIX kernel group)

and they were like "its busted" and hes like "I FUCKING KNOW THAT" and killed them

[signal zinc]

i see :3 o

[twilit smelt]

@dark shard i have new itanium lore for you

Multiple engineers we spoke with described the first boot of each new processor with great enthusiasm and detail. Havens explained that the first Itanium processors were being readied in Dupont, Washington. The Microsoft, Linux and HP/UX teams were all present on the same day in the same lab, bringing up the processor. Each team had a secure room for their software work, but worked on the hardware in a shared lab. They all successfully brought the machine up on the same day. However, the Linux team brought the machine up a few hours before the Windows team did because Intel had forgotten to initialize the keyboard controller in the firmware. Linux accepted input from the serial port, while Windows did not, forcing the Windows team to wait for keyboard controller support before booting successfully.

https://courses.cs.washington.edu/courses/csep590/06au/projects/history-64-bit.pdf

[twilit smelt]

god jackal compiler is such high quality

thats been like that for a year

the thing i hate about it is that its like

larger than the aisix kernel (my first semi-viable kernel project)

22k lines vs jackal compiler clocking in at 27k lines

but its largely speaking, junk

i mean the front end is fine, i guess, but IR generation and everything beyond is just trash

could have been simpler with higher quality codegen

so im going to redo that at some point i guess

one thing that was interesting that i considered doing the first time around but decided against bc it was too risky and i was getting bored of writing toolchain stuff was

not having a discrete assembler program

instead a jackal compiler backend will emit like a linked list of nodes, almost like an in-memory representation of an assembly program

and then this is accepted by a backend-backend which actually encodes that linked list of nodes into an object file

and the "assembler" would just be a frontend on the jackal compiler that parses directly into this linked list format and then calls the same backend-backend

i might actually do this whenever i get around to rewriting the middle- and back-ends of the jackal compiler

benefits of this:

• faster - dont have to emit to an assembly file and then call a whole other program to assemble that
• assembler gets all the jackal preprocessor goodies and various context for free
• inline assembly can be rly flexible bc of heightened context awareness between the assembler and the compiler

itd probably be more like unifying jackal and assembly into a single language, all assembly source modules would just be jackal programs that only consist of stretches of assembly (but can use jackal headers and whatever, freely, and be able to use stuff like struct offsets with nice syntax and whatever)

in practice this is what im already doing in like mintia2

https://github.com/xrarch/mintia2/blob/main/OS/Executive/Ke/xr17032/KeSpinlock.jkl

here for instance

this consists only of assembly routines but the reason its a jackal program is that the jackal compiler, if there are any #ASM [] blocks in a source file, will dump lots of .defines into the output assembly for struct field offsets for you

    subi a2, zero, 4096             // Acquire a pointer to the Prb.
    mov  a3, byte [a2 + KiPrb_Ipl]  // Load the old IPL.
    mov  byte [a2 + KiPrb_Ipl], 2   // Store the new IPL.

so i can do stuff like this

where KiPrb_Ipl is a constant automatically generated by the jackal compiler for the offset of the Ipl field within the KiPrb struct

i havent really been paying attention to my toolchain and language as being their own project

as opposed to some minor chore off in the corner for the OS project

but i probably should, because by scale its definitely the former

the compiler is probably inelegant junk bc i wasnt respecting it as its own substantial project

but im definitely not doing this soon

honestly rewriting the middle and backends to use more "modern" techniques could probably be a bachelor's thesis

so maybe ill save it for that lol

[warm pine]

if you manage to find a few novelties you could have an msc thesis even

[twilit smelt]

this part is not novel at least, im aware of at least DEC GEM as being a compiler system that did this

i do not know if gcc or clang do

[warm pine]

i note uvm in its entirety was the subject of a phd thesis despite being fairly lacking in novelty (it combined sunos style anons with a mostly mach basis, and there was the page loanout and friends, which are really iterative improvements over what was already available) so one wonders what a complete mintia could be

[twilit smelt]

a lot of fucking work

my output now is also severely kneecapped compared to what i was doing in like 2022-2023 for some reason

i thought mintia2 would go really fast bc i know exactly what to do but its been really difficult to make myself work on it

possibly because i know exactly what to do

which is boring

and depressing

probably the most uncertain thing still left to be done is something STREAMS-like for character io stacks

wait

i just figured it out

this is a graph my friend made of mintia loc in like mid 2022

u can see around the middle the slope increases quite a lot

that is where i started keeping a todo list in the apple notes app.

i think that was my secret.

and i havent been doing that this time.

[warm pine]

let me know what you are thinking abouit this, i am very eager to do streams

[twilit smelt]

i think that helped a lot bc whenever i had time and motivation i could just pop items off the front of the todo list and get them done and i didnt need to take any time to reorient myself and figure out the next move

and id add items to the end of it just whenever i thought of them and could quickly add them from my iphone in the line at panda express or wherever i was

lol

so i was offloading all the "what do i do next" thinking to times where i was too busy to work on it, leaving the times where i could work on it, free for working on it!

very small excerpt from the old mintia todo list

it was over 1000 items long

[warm pine]

what's PPTs? process page tables?

[twilit smelt]

prototype page tables maybe

idk

[warm pine]

oh that'll be it

yeah for file backed i think it's sinful

since the file tables are already pageable by proxy, since the pages they point to are evictable and can induce the removal altogether of the table

same reason i don't find paging structures describing a file's extents to pagefile - they are evictable by proxy and reconstructable, so there is a duplication if they are paged to pagefile

[twilit smelt]

my meager insights so far:

• when you write() onto a STREAMS stack, small writes can be copied onto buffers allocated in kernel space and chained onto the data queue, large writes can be pinned into MDLs and associated with a data queue item header
• when you read() from a STREAMS stack, a simple callback into the top module on the stack saying "a read is pending" along with a pointer to the IOP and whatever, is (along with probably a return value telling the IO system to do something special) enough to allow drivers to implement optimizations like directing a fancy NIC to DMA directly into your buffer or whatever

[twilit smelt]

if you had such a fancy NIC then youd probably have like

a one-deep module stack for sockets connected thru that NIC

and youd essentialy bypass a lot of the goofiness that would get in the way of just doing some freakin DMA

since its not convenient or helpful there

i mean

in that case

just dont do a streams driver

nvm its still useful to so that you can push other things on top

im unsure how to maximize the usefulness of DMA once youve done that

well for writes its obvious

you can DMA out of whatever buffer you received in the incoming data queue

this works well no matter where u are

reads are a little different

for DMA to be useful when reading a device, the read needs to be an active, explicit operation

streams is designed to faciliate like the passive appearance of data

with no specific destination

you can fudge it if youre at the top of the stack by giving some hook for awareness of the user's buffer and DMAing into that

but below that point youve got some stack of modules above you that can arbitrarily manipulate this data and isn't rly "actively reading"

although i guess you could treat it like youre always being actively read from and just have a buffer set aside to receive data by DMA at all times

[dark shard]

Thank you, itanium lore is always appreciated

[twilit smelt]

i also realized

from being reminded about amd64's rip relative stuff

i should have been doing rip relative stuff in xr17032 all along

if i ever do an xr17064 its going in there

but its too late at this point

[signal zinc]

breakingbad!!!

this is a great cover

[twilit smelt]

50th anniversary of unix is this year or last

actually it was in 2020

math is hard

[twilit smelt]

the virtual memory chapter in my architecture handbook is actually terrifying

i just read it for the first time since i wrote it like a year ago i think lol

gonna show this to the next person i see calling x86 paging difficult

"omg its so difficult to understand a little 512-ary treeeee i dont wanna build a little 512-ary tree"

design a novel software tlb refill scheme that uses nested tlb miss exceptions to walk the page table hierarchy and cache all levels by directly loading the pte from a recursive page table mappign and we'll talk!

well that bit isnt novel MIPS did that, but ive never seen it be done by nested entry into the same tlb miss handler, especially one this short

DtbMissRoutine:
  mfcr zero, dtbaddr
  mov zero, long [zero]
  mtcr dtbpte, zero
  rfe

[heady bobcat]

oh my... why did you do this?

[twilit smelt]

bc i figured out a way to set a new (unrecognized.) world record for shortest tlb miss handlers

and it was too cool not to implement

most likely someone doing one of the risc architectures realized this was possible but considered it too risky and intricate to actually attempt

https://raw.githubusercontent.com/xrarch/books/refs/heads/main/xr17032handbook/main.pdf#page=7

[twilit smelt]

the entire rationale behind the way virtual memory worked in MIPS and other RISCs was to avoid the bug prone intricacy found in CISC MMUs like that of the VAX

so this would have been counterproductive for that goal lol

this is still wayyy lighter in hardware than hardware page table walking

but substantially more intricate than like MIPS MMU logic

[heady bobcat]

alright I will try to understand this...
so

• move from control register into the zero register (and the zero register is now a general-purpose register for some reason...)
• load whatever is pointed to by dtbaddr into the zero register
• "mtcr" -- populate the TLB?
• return from exception

[twilit smelt]

the zero register is hardwired to read all zeroes until youre inside a tlb miss and then it is a gpr lol

[heady bobcat]

yea lol

just hope you don't need the zero register in the tlb miss handler

[twilit smelt]

in a real implementation itd most conveniently be a full register in the register file all along and was just stuck reading zeroes until that moment

[acoustic sparrow]

yeah radix trees are incredibly complicated!

[acoustic sparrow]

thats mildly cursed tbh

but

its better than riscv

so i cant complain

[heady bobcat]

so what if you call into non-asm code from the TLB miss handler (maybe because of an interrupt or something)?

[twilit smelt]

thereve been at least two fpga implementations of xr17032, neither of them got to the mmu (yet) but they both commented on this bit as being easy to do

[acoustic sparrow]

yeah it is

its super easy

im not saying its hard

im saying its cursed as hell

[twilit smelt]

literally all the weird tricks i thought of in the shower across a span of like 5 yrs are concentrated in how tlb misses work on this arch lol

and most of them are afaik novel

not to say theyre useful but that does mean theyre fairly interesting

for example i dont know of anybody else who "banks" the zero register like this lol

[acoustic sparrow]

tbh i'd only let you bank the zero reg for straight movs

[twilit smelt]

you dont

in theory you could undo some of the weird tlb state machine stuff and make it look like a more normal exception and then call into heftier code

but in practice you dont want to do this

this goes against the spirit of the architectural design (which is optimizing (very nicely!) for a multi level table lookup) and also probably indicates your kernel's vmm design is bad

[twilit smelt]

that mtcr is writing a tlb entry yeah

instead of doing the mips school of thought where they added a bunch of special instructions for stuff like writing tlb entries, i went with the DEC Alpha way where they have certain special "control registers" that cause actions to occur when you write to them

[acoustic sparrow]

also i just wanted to say something: you should write a jackal documentation system

so you can selfhost docs too

[twilit smelt]

the trickiest things there are the value of dtbaddr which was populated by hardware with the virtual address of the PTE you should load out of a recursive page table mapping

basically during kernel init you set the upper 10 bits to whatever the 4mb-aligned virtual address of your recursive page table mapping is

the lower 22 bits are set by hardware upon tlb misses and are calculated by literally just vpn of missed address << 2

so basically this is an address inside of a virtually linear page table array

constructed by putting the top level page table as an entry into itself

its trivial for the hardware to calculate this for you so it does (MIPS does the same)

the other tricky thing is the load

of the PTE

bc this can cause a nested tlb miss

if the page table the PTE was inside wasnt itself mapped by a tlb entry

[heady bobcat]

yeah. and that becomes messy

[twilit smelt]

this will cause nested misses up the page table hierarchy until you reach the first level that is mapped in the tlb, or the top level page table (which is permanently mapped with a wired tlb entry your kernel set up earlier)

then you load the appropriate entry out of that and return

and this returns to the original faulting instruction

which then takes another tlb miss because you didnt actually satisfy that original one you just trampled over it with nested ones

this repeats until the hierarchy has been filled into the tlb from the top down

and then you finally enter the real pte into the tlb

[heady bobcat]

this is galaxybrain

[twilit smelt]

this sounds worse than doing some other way because in a 5 level paging scheme youre going like
1 -> 2 -> 3 -> 4 -> 5
1 -> 2 -> 3 -> 4
1 -> 2 -> 3
1 -> 2
1

but the thing is

after that, those levels are in the tlb!

a future miss from the same page table will hit immediately in the bottom level page table!

so you skip looking up the upper levels entirely!

literally a bottom-up page table walk

and this pays for itself probably.

[twilit smelt]

its not as bad as this with 2 levels in any case and thats what i implement bc this arch is (boringly) 32 bit

just goes
1 -> 2
1

in worst case

[heady bobcat]

this sounds a lot like the page table TLB caching stuff from x86. so instead of storing only PTEs in the TLB, there are some entries for PML1D, some for PML2D, etc..

[twilit smelt]

internally x86 might do this in ucode or whatever

it might do this literal exact thing

i dont know

its a good idea so its difficult to imagine nobody else other than me has come up with it

[heady bobcat]

(I refuse to use the "proper" names, page table, page directory etc. since those are unnecessarily confusing)

[twilit smelt]

its also necessary if youre ruling out saving anything on the stack

which is the primary reason you cant return from an inner tlb miss to an outer one

and have to just overwrite the state for each one and then return to the original guy

and repeat

[twilit smelt]

by the way i looked at what the 64 bit RISCs with software tlb refill did, i wondered if they maybe did this

i was appalled by how lame Alpha's was in particular

the tlb miss handler on Alpha runs with paging disabled and does a full lookup of the hierarchy in physical memory

branching off to the page fault handler if it encounters a pte at any level with V=0 and whatever

this is completely undocumented btw because its hidden behind DEC-written firmware called "PALcode" and i had to find internal documents to figure out this is how they did it lol

[heady bobcat]

tbh that sounds overcomplicated. but I guess it avoids "problems" with TLB misses on the TLB miss handler itself.

[twilit smelt]

this tlb miss handler runs with paging enabled

mine

but it runs in a page youve previously wired into the itlb

it has to run with paging enabled or else it couldnt load ptes from the virtually linear page table.

[heady bobcat]

btw what happens if an interrupt occurs while your TLB miss handler is running? does the T bit being set cause the state representing the faulting instruction to be saved instead of the actual program counter etc.?

[twilit smelt]

interrupts are disabled inside tlb miss handlers

results are basically undefined if you take any exception other than a nested tlb miss

(unless youve done the aforementioned process of unwinding the weird special casing and made it look like a proper exception, switched to a kernel stack, saved registers, etc; this would involve clearing the T bit manually and stuff)

[heady bobcat]

yea but at that point just do the entire TLB miss handler and return lol

[twilit smelt]

you can also take a proper page fault if you load a pte from a page table that has an entry mapping it as "invalid" in the tlb

and thats also defined

[heady bobcat]

because of negative TLB entries?

[twilit smelt]

yes

negative tlb entries are how i avoid having any branches at all here

[heady bobcat]

so you flush the TLB when populating pages?

[twilit smelt]

yes you have to flush out an invalid entry when you make it valid in the page fault handler

this doesnt require IPIs

because each core can do it lazily

if you take a page fault and the pte has the valid bit set, just flush the tlb entry and return

[heady bobcat]

oh nice.

[twilit smelt]

this is what enabling paging looks like

[heady bobcat]

now I wonder if it would make sense to manually "pre-fault" the TLB if, for example, madvise(MADV_SEQUENTIAL) or whatever

[twilit smelt]

if u cant tell i had paging as some kind of autistic special interest for a really long time lol

its not so much anymore but i think thats because i literally ran out of things to learn about it

and ran out of cool ideas to come up with on my own

i pretty much solved software refilled tlb

this scheme is near optimal

someone at intel probably independenlt yinvented this in the 90s and its been a trade secret thing they do in ucode for decades already

lol

or maybe they patented it

and thats out there somewhere

who knows

[acoustic sparrow]

tlb refill is probably not done in software tbh

[twilit smelt]

i have heard things about the way they cache the page table levels that suggest to me they might be doing this

but i dont know

well

i guess you could just have discrete tlbs for each level and each of them is keyed by that field of the virtual address (or something in a similar vein)

and you can use that to do bottom-up lookups

in hardware

prob they do that

i like my way better though bc i came up with it.

[twilit smelt]

https://www.youtube.com/watch?v=igHQgxLvoq8

woah!

someone uploaded this a month ago

[twilit smelt]

https://www.youtube.com/watch?v=cCTDyhG_qEg

holy grail: lou perazzoli talking about Mm

@warm pine

dont ask how i found these, someone uploaded them a month ago and they have like 100 views

[warm mural]

maybe there are other NT autistics in the world

[green pelican]

the secret microsoft autism cabal

[warm pine]

this ought to be fun

cheers

[twilit smelt]

apparently xrstation isnt a fake computer anymore

@patent totem implemented enough of it on an fpga to run the bios

seen here

[patent totem]

AliExpress XRStation

[teal trench]

yoo that's epic

[dense vigil]

Damn

[heady bobcat]

does it have ACPI?

[fair jetty]

holy shit

[twilit smelt]

https://youtu.be/bA29xYcOQW8?t=376

LOL

ive now seen three different filmed instances of darryl havens shitting all over the pc floppy controller over the span of 5 years

he really hated this thing

[cunning mortar]

it kinda really sucks

so he isn't wrong at all

[twilit smelt]

its just hilarious

because ive now found three different taped presentations by this guy on the NT IO system, from 1989, 1992, and now 1994

in every single one of them he does this floppy controller comedy routine

it REALLY bothered him

https://youtu.be/19I2ffadc3Y?t=1472

https://youtu.be/mtF-X1YnIDg?t=7058

im collecting these like infinity stones

[cunning mortar]

yeah i've seen them from the other time you posted them lol

[twilit smelt]

THREE HUNDRED AND SEVENTY FIVE MICROSECONDS!

[warm mural]

That's insane

[twilit smelt]

it is, julien is v smart

[white valley]

holy shit

[twilit smelt]

i found this dumb project timeline i made in late 2021

so i updated it

[white valley]

i love how limn1k is brown

[twilit smelt]

im literally colorblind so i have a compromised idea of what colors any of those are

[white valley]

oh right, i thought you chose that cuz it's a CISC and CISC is shit

[dense vigil]

that explains the extremely weird choice of colors lmfao

[white valley]

also for future ref, these colors blend in too much

someone should make a tool for colorblind people that turns all images into a grayscale mockup of how similar they look

where the grayscale image contrasts where the hues contrast

[twilit smelt]

the criteria for selection was if i could tell apart the boundaries lol

besides that i just randomly fiddled with the color sliders like a boss.

[white valley]

you could also pick from a color palette