MINTIA (not vibecoded)

[twilit smelt]

Not a good mix

Worst semester of my academic career lol

[hybrid condor]

It sounds like a car would be useful to you

[twilit smelt]

Probably

[heady bobcat]

unlucky you

I'm very fortunate and live at a walking distance to the local university.

[twilit smelt]

WELL ARENT YOU SPECIAL

[heady bobcat]

Surprisingly no, like half of my classmates from high school will be attending the same uni and quite a few other people who will study cs there live in the same city.

(I'm so glad I live in Europe)

[warm mural]

Nah it's just public transit is shit

[heady bobcat]

And US distances being unimaginable for europeans

[warm mural]

Yea I guess

Even here the local uni is like at least half an hour using an express bus

[heady bobcat]

I'm pretty sure it's commonplace in Europe for student flats to be relatively cheap (plus good student loans etc.) so even the people attending uni from someplace else usually move into city while they study.

This is atleast the case where I live

[warm mural]

also unis dont cost you a liver

[mortal thunder]

in my country if you got a good result in the admissions exam you get uni-funded tuition

so no loans

you will have to pay for student flats though but that's fine

i was paying like 70 euro per month

though it was with 3 other people

[twilit smelt]

i couldnt do a dorm because it would have doubled my tuition

even the cheapest dorms were as much as the rest of tuition combined

like $4000 per semester

harvard has a few like $80k/yr penthouse dorms lmfao

but even at random state unis theyre pretty unaffordable

rn

when my mom went to the uofu in the 90s her dorm was $300/semester so it has gone up by over 10x

crazy

even accounting for inflation it has gone up by 10x

[twilit smelt]

imagine having enough money to pay for this for your child

[acoustic sparrow]

its supply and demand

you have rich kids that want the dorm

and so the price is adjusted accordingly

[twilit smelt]

well that dorm would be $80k anywhere

its like half of an entire floor of a high rise adjacent to the campus

its an extremely upscale apartment

[acoustic sparrow]

makes sense

[twilit smelt]

its just insane they own that and offer it

(which is, as you said, becasue almost all of their students are from rich families)

[mortal thunder]

those are for the ballers

but i guess it works better than renovating it to fit 5 groups of people for significantly less

[warm pine]

Quite sad really

How are you meant to get the student experience without halls

[mortal thunder]

the US is the land of individualism ™

[digital pivot]

and how is its condition for that price?

[mortal thunder]

Acceptable, but not great

[heady bobcat]

Your architecture's TLB replacement policy, is it like Least-Recently-Used, or do you just cycle through them, or is it something else?

And if you cycle through them, is there like a register that says to start at some index other than zero (this being used to 'pin' a number of fixed TLB entries)?

[twilit smelt]

It's FIFO

The next entry to replace is stored in the TBINDEX control register which is incremented each time

When it wraps to zero it is reset to 4 to skip the first four entries which are wired

[acoustic sparrow]

is that fixed architecturally?

or configurable

[twilit smelt]

fixed architecturally

it wouldnt be hard to have a control register that contains the value tbindex should be reset to when it wraps

which you can use to configure how many wired tlb entries

but like

bleh

[acoustic sparrow]

yea

[twilit smelt]

i already wrote an architecture manual i dont wanna add that shit

[acoustic sparrow]

it might make sense to have a non power of 2 number of tlb entries

and make the wired ones at the top

because that removes a branch (inisgnificant but yk) and is easier to impl in hw probably

[twilit smelt]

what branch

[acoustic sparrow]

the branch on resetting to zero in the emulator

[twilit smelt]

ah

i mean the way its designed is meant to make the system software agnostic to the tlb size

[acoustic sparrow]

fair

[twilit smelt]

mips made a mistake imo where they baked in the assumption of 64 tlb entries and they had to have exactly 64 forever because everybody hardcoded loops of 64 iterations for tlb management

[acoustic sparrow]

lol ofc

[twilit smelt]

the way i set the wired entries is to manually write one of their indices to tbindex

before writing tlb values to the tbpte control register, which causes a write to the tlb entry currently pointed to by tbindex, and advances it by 1

your way would require the software to know what index the "top" of the tlb has

ie the size

unless i added some other special way to set those entries

also theres a very simple and highly effective improvement to FIFO called NLU or not last used

where basically its just fifo but if an entry was the last one referenced, you skip replacing it

and replace the next one instead

Alpha used this

i could also use this possibly but its more logic

[twilit smelt]

theres another improvement to NLU where you have a fifo buffer that contains the like 4 last recently referenced virtual page numbers or something rather than just one

and you can compare the vpn of the tlb entry youre about to replace associatively with all the vpns in that buffer and skip it if its any of them

this is relatively easily implemented as well

[twilit smelt]

the nice thing about the way i did this though is that its completely transparent

so i could add that and like

system software would not care

[acoustic sparrow]

makes sense

[twilit smelt]

MIPS actually did random replacement

of the tlb

"random" was really the low 6 bits of a cycle counter

[heady bobcat]

That seems like it is in theory possible to achieve "funny" outcomes by generating TLB misses at very precise moments in time...

[twilit smelt]

fifo with 4-entry nlu buffer probably performs better than random in more situations

thats my intuition

i havent actually benchmarked that

[teal trench]

is your tlb fully associative?

[twilit smelt]

Yeah

[teal trench]

hm i suppose it wouldn't make much sense for it not to be with the tbindex register stuff

unless it was like 64-way set associative :^)

[twilit smelt]

Finally doing way overdue toolchain work

I like copypasted a big chunk of the jackal frontend (in particular the preprocessor and the lexing essentials) for the assembler and build tool

That was a mistake I should have common-ized it

Doing that rn

I plan on reusing the exact same lexing infrastructure for the mintia2 command prompt/scripting language later as well

Which will be a fourth consumer of this same thing

So it best be common

This has been bugging me and making me ashamed for a year and a half now

The key to (eventual) excellence is guilt

[lucid umbra]

thinking about making a channel for coyote

since it would fall under the tools category

i think

[twilit smelt]

i expanded the number of signals to 64

to make way for the real time signals

i reserved a few for my own use

[twilit smelt]

notes: OS_SIGNAL_KILL is a special signal that causes immediate death. OS_SIGNAL_TERMINATE_PROCESS and OS_SIGNAL_TERMINATE_THREAD are normal signals that are used to initiate the runtime library's graceful termination.

planned flow of graceful process termination:

1. process calls OsTerminateProcess(process, status). this sets the termination status of the process to the given value and then sends OS_SIGNAL_TERMINATE_PROCESS to either the signal handling thread of the target process, or the current thread if it is part of the target process.
2. runtime library runs in the context of the signaled thread. it sends OS_SIGNAL_TERMINATE_THREAD to all of the other threads.
3. the signaled thread performs its own graceful thread termination without re-signaling itself.
4. runtime library runs the process-wide on-exit callbacks.
5. runtime library sends the process OS_SIGNAL_KILL to cause final termination.

graceful thread termination:

1. thread calls OsTerminateThread(thread, status). this sets the termination status of the thread and then, if the process has only one thread, it merges into the OsTerminateProcess(thread^.Process, status) path and that proceeds as above. otherwise it sends OS_SIGNAL_TERMINATE_THREAD to the given thread
2. runtime library runs in the context of the signaled thread. it runs the per-thread on-exit callbacks.
3. runtime library sends the thread OS_SIGNAL_KILL to cause final termination.

something like this

this is fairly fragile

for example you can kind of easily cause a race where the OS_SIGNAL_TERMINATE_PROCESS is never sent by having two threads in the process and having them terminate eachother in quick succession before the other has a chance to exit properly

theres a way to fix this i guess

which is to only have OS_SIGNAL_TERMINATE_THREAD and the process termination part is done when an atomic count of active threads (maintained in userspace) is decremented to zero from inside its handler

at which point you call the graceful process exit stuff

which is just a normal function

hmm yeah thats probbably the good move

i can make it so that if you direct OS_SIGNAL_TERMINATE_THREAD to a process object, it gets broadcasted to all threads atomically

and also prevents any new threads from being created

as OS_SIGNAL_KILL does

[twilit smelt]

i wonder if this signal should also be unmaskable

[twilit smelt]

it is now

[twilit smelt]

but it probably shouldnt be

it is no longer

@warm pine ignore the above its already outdated

but basically i extended the number of signals from 32 to 64 to make way for the real time signals and i reserved a handful of them for graceful termination

OS_SIGNAL_TERMINATE_THREAD in particular

the OsTerminateProcess and OsTerminateThread APIs now just set the return status of the process/thread and then send it OS_SIGNAL_TERMINATE_THREAD

rather than OS_SIGNAL_KILL

sending it to the thread causes that one thread to execute the signal handler, do per-thread exit processing (like TLS teardown), then it sends itself OS_SIGNAL_KILL which actually kills it for real

[warm pine]

OsTerminateProcess can operate on another process too and set its exit status?

[twilit smelt]

you can still send OS_SIGNAL_KILL directly if you want to be mean and then userspace gets no chance to do anything

yeah

[warm pine]

i suppose the unix subsystem will have alternative arrangements available

[twilit smelt]

also if you send OS_SIGNAL_TERMINATE_THREAD to a process itll specially broadcast the signal to all of the process's threads, and the last one out will do the userspace side process exit stuff (like running atexit() type things)

and thatll be the preferred first-line way to terminate a process

it also atomically prevents any new threads from being created in the process like OS_SIGNAL_KILL does

so all the ones that are there will receive this signal and no new ones can enter the picture afterwards

[warm pine]

will this behaviour be maskable?

i know at least one program whose clean exit path requires it to spawn a new thread

[digital pivot]

what happens if a process is a part of a job-like grouping ?

[twilit smelt]

does it do that in an atexit() or in a SIGTERM handler or what

[twilit smelt]

nothing special

if you send OS_SIGNAL_KILL to a job it atomically prevents any new processes from being created within the job though

[warm pine]

it masked SIGTERM but detected it in its main event loop with kqueue EVFILT_SIGNAL (= turns signals into events you can multiplex, as if with poll/select), i can't remember the program

[twilit smelt]

this signal is not equivalent to SIGTERM

so thats not an issue here

it would receive SIGTERM and then do its thing with the thread and then when it calls exit(0); or whatever thatll wrap a call to OsTerminateProcess(OS_CURRENT_PROCESS, 0) which sends this special signal

and does the native mintia userspace termination

[digital pivot]

can job prevent its processes from being terminated outside of the job termination itself ?

[twilit smelt]

wdym

[digital pivot]

or it doesnt work like that

[twilit smelt]

SIGTERM is a whole other signal is what im saying basically and isnt special at all

OS_SIGNAL_TERM in mintia parlance

[digital pivot]

as i understand it you can either kill the entire job (kill all its processes) or kill a separate process that is part of the job

[twilit smelt]

yeah you can do either of those

[digital pivot]

i was surprised reactos doesnt support jobs

[twilit smelt]

my jobs arent NT jobs

theyll be used for session control and stuff in the native mintia userspace

[digital pivot]

well it has https://github.com/reactos/reactos/pull/7500 which is draft since 2024

[twilit smelt]

NT jobs arent appropriate for that use case

[digital pivot]

yeah looking at this nt jobs are pretty weird in some cases

[twilit smelt]

my jobs are more like mica's jobs

[warm pine]

this is entirely reasonable then, i think it's a nice touch to have the termination process explicated in this way

[twilit smelt]

btw broadly speaking you will probably not ever send OS_SIGNAL_TERMINATE_THREAD to another process, although you can, it will be mostly used internally by the runtime library (osdll) for like thread termination. you would be more likely to send a signal like SIGTERM which will invoke the thing's handler or if there is none itll just exit which will OsTerminateProcess itself which will then send the OS_SIGNAL_TERMINATE_THREAD which will ultimately cause it to send itself OS_SIGNAL_KILL

and then it dies fr.

[heady bobcat]

and OS_SIGNAL_KILL doesn't set the exit status but leaves the one stored by OS_SIGNAL_TERMINATE_THREAD?

[twilit smelt]

OS_SIGNAL_TERM: nicely ask it to die
OS_SIGNAL_TERMINATE_THREAD: ask it a little less nicely, probably inappropriate to ever send from outside of the process
OS_SIGNAL_KILL: im not asking. just die.

OS_SIGNAL_TERM and OS_SIGNAL_KILL directly correspond to SIGTERM and SIGKILL

[twilit smelt]

sending a signal doesnt do anything to the exit status by itself no

at least not at the native mintia level

the linux subsystem implementation of like the kill system call or something might twiddle the exit status to get the like 128+signal num exit status that you expect, or wherever is appropriate to do that

things like that

but OsSignalProcess and OsSignalThread dont

also OsTerminateProcess might write a value into the process's PEB saying "the entire process is terminating" so that the threads executing the OS_SIGNAL_TERMINATE_THREAD handler dont waste time doing TLS teardown and stuff

since the process's address space will be completely torn down anyway that would just be a waste of cycles to do it explicitly

[digital pivot]

will your handle table be pageable like in windows ?

[twilit smelt]

yeah it was in old mintia

still wil be

[twilit smelt]

I may

Completely remove the process termination aspect of this and use it solely for thread termination within a process

Yeah I'm over complicating this I just need to figure out what will be needed for pthreads cancellation

Do that and no more no less

[twilit smelt]

Turns out all i need for pthreads cancellation is a normal ass internal signal to use for it

and a way to defer that signal until a "cancellation point"

i already have that in the form of the DeliverOnWait signal mask

[twilit smelt]

i was wondering why the page zeroing thread was causing firework stutters

turns out i forgot to make timeshared threads preempt idle priority threads (in fact thats the only thing they can preempt)

woopsie doodle

[twilit smelt]

lol i have an incredibly interesting example of how tiny things in scheduler code can have massive effects now

[mortal thunder]

The kernel should handle cleanup for everything, extra per-thread exit processing is only a nicety

Otherwise, what happens if a program crashes, or exits improperly?

[twilit smelt]

reposting this here

@mortal thunder i found a pathological case caused by a 1+ year old scheduler bug

(pay attention to the tiny firework particles)

[twilit smelt]

it was this

[mortal thunder]

so before you add the condition the fireworks stutter like crazy

[twilit smelt]

yeah lol

its funny how 2 lines can change the behavior so much

[mortal thunder]

what function is this? one that emplaces a thread on a queue and prepares it for execution?

[twilit smelt]

yeah

[mortal thunder]

well logically you wouldn't preempt a normal thread to make way for an idle thread

lemmie see if i have something similar

hmm no, i insert threads into the scheduler queues manually in several places

[mortal thunder]

no, but I had a similar problem actually

when a thread is waking up from a wait I wasn't preempting it if its priority was higher than the current thread's

which is what this does

[twilit smelt]

whats the point of the two timers

[mortal thunder]

WaitTimer is used to handle timeout for wait operations

and SleepTimer is used by OSSleep (my sleep system call)

[twilit smelt]

you can just use the same timer for both

[mortal thunder]

right

i could just wait on a never signalled object and have the timeout as the duration

[twilit smelt]

or wait on the timer directly

is what i do i think

[mortal thunder]

but tbh the wait timer has a bit of extra logic because it fires a DPC to wake up the thread manually

meanwhile when waiting on the sleep timer, the timer code wakes up the thread automatically

[twilit smelt]

i realized disabling migration for the current thread only requires like

5 instructions

because i dont need to take any spinlocks or anything to change the Pinned boolean in the thread struct

since its only read while the thread is on a ready queue

and if its currently running then... its not on a ready queue

so i can just set Pinned

FN KeControlMigration (
  IN migrate : UWORD,
) : UWORD
    
  // Disable thread migration for the current thread.
  // Returns the old pinned state.

  thread := KeCurrentThread ()

  // No lock is needed because Pinned is only read while the thread is on a
  // ready queue. Since it's running *right now*, it can't be on a ready
  // queue!

  oldpinned := thread^.Pinned
  thread^.Pinned = migrate

  RETURN oldpinned
END
+-+
+-+
KeControlMigration:
  subi t0, zero, 12292
  mov t0, long [t0]
  mov a3, byte [t0 + 110]
  mov byte [t0 + 110], a0
  ret 

that's a nice thing to be able to do quickly

[digital pivot]

did you understand this while looking at the code

i mean how do you debug your scheduler

<#osdev-misc-0 message> another kernel having ke/ki distinction

[twilit smelt]

i mean there was just some weird stuttering and i thought about why that could be

i was fairly certain i correctly implemented timeshared threads preempting idle priority threads

but it would be a really good explanation so i double checked anyway and found that

and thats what it was

[twilit smelt]

btw i implemented the thing i was talking about for page zeroing

the per-cpu pages

https://github.com/xrarch/mintia2/blob/main/OS/Executive/Mm/MmQuickMapping.jkl#L66

// Failed to trylock a page. Increment a counter and lock whatever
// it lands on.

i = block^.NextIndex & (MI_QUICK_PER_CPU - 1)
block^.NextIndex = i + 1

IF KeGetLockOwner ( &block^.Locks[i] ) == thread THEN
  // The lock is held by the current thread, so we can't block on it or we
  // will self-deadlock. This can happen if this thread previously
  // acquired this quick page and then took an APC which we are running in
  // now. Luckily the solution is easy - we can just skip ahead by one and
  // lock that one instead.

  i = (i + 1) & (MI_QUICK_PER_CPU - 1)
  block^.NextIndex = i + 1

  // It should not be possible for this lock to also be held by the
  // current thread, because thread execution can only nest a maximum of
  // two levels deep (normal execution and KAPC), we're in the second
  // level, and we already skipped the lock held by the first level.

  KeAssert ( KeGetLockOwner ( &block^.Locks[i] ) != thread )
END

i love APCs

(this is an alternative to just disabling them during page zeroing and stuff which i didnt want to do)

this ended up being very very fast

almost all of the "function calls" there are macros (ie, inlined)

the only actual function calls are KeTryAcquireLockExclusive, KeFlushMyTbAddress, KeReleaseLock

@warm pine the current thread self-pinning reduced to this

#MACRO KeControlMigration ( thread, pinned ) [
    // Control whether the current thread can migrate to another CPU.
    //
    // We take a thread pointer to avoid having to load the current thread
    // redundantly, but it's not allowed to be anything other than the current
    // thread. This is because we don't take any spinlocks while disabling
    // migration, relying on the fact that the Pinned field is only accessed
    // from other contexts while the thread is on a ready queue, and if it's the
    // currently running thread, it ain't on a ready queue.

    NOTHING (thread)^.Pinned

    KeAssert ( (thread) == KeCurrentThread () )

    NOTHING (thread)^.Pinned = pinned
]

v fast

so i didnt lose much compared to the old style zeroing where id disable preemption while using a single per-cpu virtual page

[warm pine]

i am having a moment, why the first NOTHING thread^.pinned ?

[twilit smelt]

thats just for the "return value"

it returns the "old" value of Pinned

pretend the NOTHING isnt there you dont wanna know

[warm pine]

i should like to know

[twilit smelt]

its just evil jackal lexer stuff

jackal grammar stuff i mean

[warm pine]

at first i thought it was like Modula-3's EVAL (evaluate an expression in statement position) but seemingly not

[blissful smelt]

@twilit smelt didn't know mintia corp made merch

pretty cool to see some in the wild

didn't know you guys were japan based

[warm mural]

i have this on my desk

[teal trench]

no way asahi linux x mintia collab

mintia for apple silicon confirmed

[heady bobcat]

what if you have MI_QUICK_PER_CPU number of tasks each holding one page and blocking on the next?

e.g. (if MI_QUICK_PER_CPU=4):
task 0 holds index 0 and blocks on index 1
task 1 holds index 1 and blocks on index 2
task 2 holds index 2 and blocks on index 3
task 3 holds index 3 and blocks on index 0

this is extremely unlikely but possible to happen, and it causes deadlock

[heady bobcat]

this can happen if e.g.

TASK 0
MmUseQuickPte(...)
→acquires index 0
→interrupt! reschedules
TASK 1
MmUseQuickPte(...)
→acquires index 1
→interrupt! reschedules
TASK 2
MmUseQuickPte(...)
→acquires index 2
→interrupt! reschedules
TASK 3
MmUseQuickPte(...)
→acquires index 3
→interrupt! rescheudules
TASK 0
→APC!
 →MmUseQuickPte(...)
  →blocks on index 1
TASK 1
→APC!
 →MmUseQuickPte(...)
  →blocks on index 2
TASK 2
→APC!
 →MmUseQuickPte(...)
  →blocks on index 3
TASK 3
→APC!
 →MmUseQuickPte(...)
  →blocks on index 0, deadlocked

[heady bobcat]

@twilit smelt how do huge pages work under a software TLB refill scheme like yours?

[twilit smelt]

Bleg

I think you're right

I probably need to just disable APCs sadly

that fixes it up

that or have a different pool of PTEs for if you're currently executing an APC versus not

so if you're executing an APC and you contend on this lock you'll only ever contend with someone else who is also executing an APC and so they'll definitely be able to come in and free it up

Idk why I'm obsessed with being able to deliver APCs in that section

I dislike blocking anything anywhere when I don't have to it makes me feel congested irl

especially during smth as lengthy as zeroing a page

[mortal thunder]

hey i have another question, where does mintia1 store the file and the offset within said file in the page frame struct, or how does it obtain this data?

[twilit smelt]

Another alternative is to forbid APCs from ever calling this at all lol

But I dislike that that limits their usefulness for no good reason

[twilit smelt]

I believe PFN entries for file cache pages contained an offset within the file and a pointer to the FCB

[heady bobcat]

One thing that you can do is that, if the current task is holding a page table entry and it gets interrupted by an APC, that APC can modify the PTE and use it temporarily only to later restore it.

Pseudocode:

FN MmUseQuickPTE(
    a bunch of arguments
)
    thread := KeCurrentThread()
    ptep := thread.^InUseQuickPTE
    IF ptep != NULL
        save old pte
        write pte and flush address
        do stuff with mapped page
        restore old pte and flush address
    ELSE
        the current code, except you set thread.^InUseQuickPTE after getting a PTE and set it to NULL afterwards
    ENDIF
END

[mortal thunder]

i don't think i saw those in MmInternal.h but i'll take another look

https://github.com/xrarch/mintia/blob/main/OS/OSKernel/Memory/MmInternal.h#L3 yeah I'm not sure which one it is

nvm, it says it at the top, IOiPageFrameEntryCache

[twilit smelt]

https://github.com/xrarch/mintia/blob/2e60b083e5929e016f300d6e97f5442f891b07e1/OS/OSKernel/IO/IOInternal.h#L61

This is what you want

[mortal thunder]

okay, neat, so i wanted to do a similar thing but i was skeptical (because i couldn't find it here nor in the NT kernel's structs)

thanks

[twilit smelt]

That is very clever

[twilit smelt]

Why are you smarter than me. I hate you!!!!

I think you've found the correct solution yeah

[heady bobcat]

you need to disable APCs for a short period however, because if you take an APC after getting a PTE but before storing it in the thread, it's the same issue as before

[twilit smelt]

Dragonfruit was so beautiful

[twilit smelt]

The idea of reusing the PTE out from under the thread while it executes the APC and then surreptitiously replacing it like nothing happened has rewired synapses in my brain

That's incredibly funny

I will do it immediately

That's some goofball shit I'd find in NT and be mesmerized by and tell people here and they'd go "that's fucked up."

"you're not supposed to use PTEs like that. where's your map_page function. what is this."

[mortal thunder]

this wouldn't be the only place that works like that

[twilit smelt]

half of why I write a kernel is to be contrarian it just delights me idk why

[mortal thunder]

in my kernel, when an APC interrupts a thread while it's waiting, the APC executes and can run a wait of its own, and when the APC exits, the interrupted thread will silently rerun the wait

[twilit smelt]

@heady bobcat you're becoming a very good contrarian

[twilit smelt]

Yeah this is a thing

I also have like 50 retry cases for the blocking mutexes

[heady bobcat]

@twilit smelt did you miss this message?

I'm somewhat interested in computer architecture design, and so I thought a bit about software TLB filling, but I don't see how you can, in an obvious way, implement huge pages with the bottom-up page table walking that you do

[twilit smelt]

I don't know. I know MIPS did bottom up walking as a fast path and it had huge pages so you might want to see how they did it

I have ideas though

[heady bobcat]

oh my god I think I have a possible solution which works well with bottom-up walking

so let's say that you get a TLB miss for address X

the TLB miss handler will now look at PTE PTE_BASE + ((X & linear_address_mask) >> PAGE_SHIFT)

that faults

so now the TLB miss handler looks at PTE PDE_BASE + ((X & linear_address_mask) >> PAGE_DIRECTORY_SHIFT)

it sees a PTE which has some bit set to indicate "this is a huge page"

ah yeah and that's it, because it populates the TLB with a huge page and then just returns

except no

because the fault address is wrong

[twilit smelt]

Yeah that was basically my idea except the TLB entry would contain the wrong VPN yeah. So basically here's what I would do

[heady bobcat]

you can get the correct virtual address by doing some math on the address of the level-1 PTE which doesn't exist, which is the address of the nested TLB fault

and you do this math if you see a "huge page" bit

[twilit smelt]

The thing is, you can easily calculate the base of the huge page from the virtual address of the PDE

[heady bobcat]

exactly

[twilit smelt]

It's just bit shifts

So the CPU could help you by doing that and using it as the matching VPN in the TLB entry whenever you write a PTE that has the huge page bit set

so the software TLB miss handler is unmodified and remains branchless

This would just be a bit more logic in the pursuit of making a "preferred page table scheme" efficient

It still doesn't stop you from using arbitrary page table formats (or even no page table at all) because you can still trick the cpu into writing the right thing for huge pages by fiddling some control registers to make them look like a huge page tlb miss

when you write the entry

so this would be fine

[twilit smelt]

At what point does this start being more on chip logic than if you just did hardware page table walking and a multi level tlb lmao

I've been led to believe that hardware page table walking is a behemoth of logic and that simple state machine type stuff for accelerating software tlb refill is like only 10% as large

[twilit smelt]

But this is getting to be a lot

Along with the other special stuff

[heady bobcat]

It's not really that much, I don't think

[marble socket]

yoo hows mintia going

[heady bobcat]

especially if you are like any "modern" architecture and can do this stuff in µcode

[marble socket]

yo fucking balling

[heady bobcat]

because if you have µcode for the TLB fill instruction, you will naturally just put the hugepage logic in that, and it costs zero on chip logic because you already had that on chip logic used for other stuff

[twilit smelt]

@heady bobcat should I credit you for the cool pte trick

in a comment

and how should i credit you

[mortal thunder]

as a fun fact i had previously done something similar https://github.com/iProgramMC/NanoShellOS/blob/40883589a7d7a03be626624488bb7b59ff05e6d2/src/mm/base.c#L70

(this is an outdated commit from 2022)

[marble socket]

your smarter then yourself?

😭

[twilit smelt]

what about my smarter

[twilit smelt]

nevermind i decided that was more complicated than its worth and im just going to disable APCs

[marble socket]

there is 2 wills

crazy

[twilit smelt]

where theres a will theres another evil shadow will

[marble socket]

https://tenor.com/view/deltarune-lebron-james-chaper-3-knight-gif-6631766238794640830

[twilit smelt]

evil idea: an architecture that automatically does tlb invalidations when you write new ptes into a virtually linear page table mapping

eviler: it broadcasts this to all other cores as well

so now you dont need to do any explicit invalidations which makes system software slightly nicer at the expense of a lot of things being way slower for no reason

[blissful smelt]

I saw HPPA do that for tlb flushes

unless I misread

[twilit smelt]

i do not believe it does this

[blissful smelt]

oh yes not to that extent

kablooey

[twilit smelt]

the thing is, this part is very easy

you can tell it where the base of the virtually linear mapping is (with the base of the region aligned to its entire size) and it can easily use the pte address to figure out which virtual address needs to be invalidated

that could even be nice

[twilit smelt]

this is the bad part

[blissful smelt]

what is the thought process behind designing an architecture where tlb purges are automatically broadcasted though

[blissful smelt]

this seems rather odd

[twilit smelt]

you can avoid a TLB shootdown IPI

and it opens opportunities for hardware-level optimizations

in the future

this has become a thing once more

the current amd chips have this now

and the intel ones might too?

[blissful smelt]

oh yeah I saw that on some of their higher end server chips they have that I think

doesn't tlb shootdown tend to be kind of expensive and thus get batched in software

pretty fun stuff

[twilit smelt]

yes and youd need a way to do the same with the hardware broadcasted tlb shootdown

for it to be any good

i can imagine you could have two instructions

one queues up a virtual page for invalidation on all cpus

without waiting for the invalidations to complete

and the other waits for them to complete

something like that

[twilit smelt]

It's time to start working on a new cached interpreter core for the emulator

[shut belfry]

can you make pins?

I don't know if people can pin stuff in their own threads

would be useful to track progress

[mortal thunder]

No, they can't

[shut belfry]

damn

well, anyone reading this, ping me and I'll pin shit for you

[heady bobcat]

if you would've done this instead of disabling APCs, something like "cool PTE trick proposed by dbstream" would be fine

but it literally doesn't matter lol

[blissful smelt]

https://en.wikipedia.org/wiki/Asynchronous_system_trap what's the main difference between this and APCs

they sound pretty similar

XNU has these

[twilit smelt]

XNU's are only executed on return to usermode I believe

[warm pine]

historically they had more significance but now they are in in the surviving BSDs basically just a name used in disparate ways

[twilit smelt]

Also I implemented it but then it was kind of a big slow mess and necessitated an extra field in the thread structure and I was like this probably isn't worth it

[warm pine]

in freebsd the deferred invocation of the scheduler following an interrupt is described as such, in netbsd signal delivery on some arches is described as such

[twilit smelt]

Nooooooo I can't deliver a kernel APC until I'm done zeroing a page noooo :((
Who cares.

[warm pine]

the concept is woolly and unclear, while nt's DPCs and APCs are considerably less so

[twilit smelt]

The action of the page zeroing thread will make it unlikely that a random thread has to synchronously zero out a page anyway

[twilit smelt]

ill probably only have energy and time to do one major thing between summer and fall semester and it can be either of the following:

1. rewrite the emulator core as a cached interpreter (should see 2-3x+ speedup)
2. rudiments of the mintia2 io system

which one.

[tardy harbor]

2. 👍

[night needle]

2

[acoustic sparrow]

2

[safe hollow]

2

[marble socket]

2

😋

[tardy harbor]

if you don't like that everybody's saying 2? do 1 instead 👍

[twilit smelt]

#1 it is

[twilit smelt]

C is so clunky

jackal >>>>>>>> C

in jackal you can just use a pointer to an undefined type in another type definition and it doesnt care as long as you define the type by the time you actually dereference it in code

i forgot in C you have to do a forward type declaration

i dislike that

[twilit smelt]

@acoustic sparrow would it be too slow if upon each basic block execution i unlink it from an lru list and move it to the front

so i can reclaim from the tail of the lru list when there are more than the maximum number of active basic blocks

(which i think will be about 4096)

[twilit smelt]

it's a circular list ofc so this is branchless

[mortal thunder]

no you don't necessarily have to

if you use the struct keyword for your types you can get away with it in some cases

so you can just do
struct something* somefunction(int a, int b);

but you can't do struct something* somefunction(struct someotherthing* a, int b); so that struct someotherthing is definable later because it will see two different instances of struct someotherthing

you USUALLY have to and you DEFINITELY have to if you use typedef types, but this isn't a hard requirement

[twilit smelt]

i find C way more obtuse than jackal in general

but thats kind of not surprising considering C was made by someone else and jackal was made by me specifically for my preferences

especially macros and stuff

i mean jackal macros just own compared to c macros

theres no contest

theyre hygienic AND you can just define them as a block of code and not need the goofball \ at the end of every line

[acoustic sparrow]

probably

depends on the size of the BB

i'd implement a multi-level hashtable

[twilit smelt]

it'll be very simple to benchmark both ways so I'll just try

[acoustic sparrow]

fair

my guess is that cache effects will make a two level hashtable faster

[twilit smelt]

Let's pretend I don't know what you specifically mean for a second. What do you mean

Oh I know what you mean

I know a very natural way to do the hash functions too

The first level table can be the first few bits of the VPN and the second can be the first few bits of the pc (other than the first two which are always zero anyway) and this will make it faster to invalidate pages at a time of cached blocks I think

[acoustic sparrow]

is that very natural way to use the crc32 instructions that allow you to compute a crc32 on 64 bit inputs in a single cycle?

[twilit smelt]

inchresting!

[acoustic sparrow]

actually its 3 cycles

maybe its less on shit thats newer than m1 lol

but you can issue a new crc instruction every single cycle anyway

and then you can just implement a SIMD probed hashtable or w/e

[twilit smelt]

These sound like great PRs for you to send me once I have the basic functionality of the cached interpreter working

What do ya say

[acoustic sparrow]

yea maybe

[acoustic sparrow]

fwiw shuffling bits like that is probably quite slow

[twilit smelt]

you're quite slow

jk you probably know more about modern performance engineering than me

i would trust your performance PRs whole heartedly

[acoustic sparrow]

all i know is how to read microarchitecture instruction tables, i saw a conference talk by matt godbolt, and i know how to use perf :^)

[twilit smelt]

my C is probably full of UB and bad practices and syntax weirdnesses that someones gonna scream at me for and im gonna say ok idc

consequences of learning low level dev with made up programming languages

and not really practicing C much

i think this emulator is the only non-trivial C code ive ever written

i really honestly dont like C

just by vibes i think it has terribly clunky vibes

[twilit smelt]

this isnt even a joke i literally learned how to do this by writing shitty compilers in lua for languages that ran on fake computers

i happened to learn C's syntax by osmosis from reading kernel source code as well

but im not a particularly good C developer

i think i tend to write what would be considered very shitty C and i do it with a half-assed jackal code style that i dont even consistently use and probably just makes it worse lol

ive wanted to rewrite the emulator for a while (in its entirety) and make it be a prettier C style

cuz im not proud of the current state of its codebase

[digital pivot]

you talk about xremu repo, right?

[twilit smelt]

yeah

[acoustic sparrow]

what is an scache?

is it for cache coherence stuff?

[twilit smelt]

basically an l2 cache that i only simulate insofar as it serving a purpose of containing some cache coherency info yeah

[acoustic sparrow]

ah

[twilit smelt]

i dont actually simulate the contents of that one

[acoustic sparrow]

fwiw you can implement 16 way cache probing very efficiently using simd stuff

[twilit smelt]

just the state of its lines

[acoustic sparrow]

basically, you store a simd vector of the 8 bit truncated hashes, and then use cpu instructions to find bits that are set

[twilit smelt]

Scache is an old timey phrase for l2 cache which stands for "secondary cache"

another name for it was Bcache, "backup cache"

[acoustic sparrow]

ahh

[twilit smelt]

since youre a C expert you should get really autistically hyperfixated on fixing up the xremu codebase until its shiny and runs at 500 MIPS (it does like 30-50 MIPS on my m1 mba rn)

(will probably do at least 60-100 after this overhaul of the emulator core)

[acoustic sparrow]

tbh its mostly that i know a bunch of generic tricks

[twilit smelt]

well you probably also know the C language better than me and can fix all the UB and bad style practices that im sure are everywhere

[acoustic sparrow]

and i know that if you throw a jit at it, it will go really fast :^)

it doesnt look that UB

have you tried running it with -fsanitize=undefined?

thats a good first step lol

[twilit smelt]

i basically assume a little endian host everywhere is the only thing im majorly aware of but thats "on purpose" and is explicitly stated in the readme

[acoustic sparrow]

yeah thats a good assumption anyway

[twilit smelt]

its "on purpose" in that i didnt care enough to pollute it with endian conversion macros so that some idiot can run it on ppc as a novelty trick

when i already thought it looked ugly enough

i also dont have a way to test that it would work on a big endian host anyway even if i did support that

i dont like supporting stuff i cant test

i guess i could build it for mac os 10.4 or something and run it in qemu ppc lol

idk

this could actually be way more than 2-3x improvement

im noticing how many bit shifts and stuff this is eliminating

im currently in the extremely tedious part where i write the decode and execute routines for like 100 different opcodes

[acoustic sparrow]

"fun"

[twilit smelt]

i can tell you one thing

my nice unrolled memset

is going to go HYPERSPEED

that kind of thing will benefit the most from this lol

[acoustic sparrow]

like in mintia?

[twilit smelt]

yes the jackal rtl memset

and memcpy

[acoustic sparrow]

tbh youd get more performance if you would make it a slow memset, and then got the emulator to recognize that memset pattern and perform a fast host memset :^)

but thats cheating

[twilit smelt]

i was also thinking of having each basic block contain a cache for like the most recent tlb lookups performed by instructions inside that basic block

maybe 2-4 of them

replaced in fifo order

[acoustic sparrow]

i think i might have mentioned this

its called an "inline cache"

because its inline with the code

[twilit smelt]

well i wasnt gonna inline it with the code i was gonna try to pack the cached instructions closer together by putting it in the basic block's header instead

hopefully i can get better dcache that way

[acoustic sparrow]

yea same thing

its close to the code, so basically inline

[twilit smelt]

this probably will need to be optional otherwise the tlb simulation becomes useless for benchmarking tlb misses

because the effective tlb size will become like.

ludicrously huge

and tlb misses will basically just not happen

after the first time

ill probably make it something you can compile out

the sad thing is im going to end up having to dump basically just the entire basic block cache whenever theres a tlb or icache invalidation

but i think ive done a good job of batching these things so that might not be too bad

i also think the decode will be so fast that even in the contrived worst case where youre doing the decode part for literally every instruction (jumping around wildly or something) itd still be 25-75% as fast as the old emulator

and in a more realistic worst case it may just be on par

so im not actually that afraid of dumping it sometimes

this is all just speculation until benchmarked though

something i really enjoy is the fact im moving some stuff i used to check every instruction into an outer loop and only checking on basic block boundaries

like checking for a pending interrupt

thatll be really nice

who knows what other optimizations this will let me cook up as well later

thats why im thinking it may end up being even better than a 2-3x improvement

if i finish all this and its SLOWER than previously then im going to no longer be alive

i doubt thats possible though even if i do it as stupidly as i can

i do fear it though

[acoustic sparrow]

why?

oh right you need to redo a lookup

[twilit smelt]

the basic block cache is virtually tagged

and even if it were physically tagged

[acoustic sparrow]

you can make it VIPT, no?

[twilit smelt]

its full of inline tlb caching

[acoustic sparrow]

just cache the index

and then do a validating lookup

[twilit smelt]

this ruins one of the benefits which is that i dont need to consult the tlb on basic block lookups

[acoustic sparrow]

yea fair

[twilit smelt]

i have a feeling the gains from that will outweigh the losses from dumping the cache each time theres a tlb invalidation

but we'll see

thats another thing itd be fairly easy to just benchmark both ways and select the better one

im going with my intuition first though

[acoustic sparrow]

you could also try playing clever tricks to avoid dumping the icache in the common case

[twilit smelt]

also im SO glad my past self decided to go with an incoherent icache that needs explicit invalidations lol thats gonna make this so much easier

[acoustic sparrow]

yeah lol

[twilit smelt]

if i needed to efficiently track when i write into the middle of a cached basic block and then invalidate it thatd be so evil

[acoustic sparrow]

for example, if the guest only writes to the architecturally preferred autoincrementing tlb index, then you can check that the tlb has not moved past any inline cache entry and thats it

[twilit smelt]

its funny this evolved from me thinking that i could cache a function pointer to the instructions' implementations as part of the icache (which would have been a like 15 line change with maybe 50% gains) to rewriting the entire emulator core

meanwhile mintia2 io system sits neglected

i have been doing tons of performance work though this hasnt been a waste of time at all, its visibly improved from like 2 months ago especially after today's cached interpreter work pans out

i think these long tangents of tidying things up rather than implementing new features that i tend to go down are good

terry never did that and thats why templeos is the way it is.

andreas kling also didnt. see serenityos.

tangents = good

[mortal thunder]

to get to that point you'd probably have to start JITting at this point

[twilit smelt]

i think you need to maintain constant shame in everything you do that compels you to go improve it eventually otherwise your software is doomed to suck

if someone gets too arrogant in their head and loses their shame theyre done for

dave cutler is probably constantly deeply ashamed and insecure about all the code he's ever written and thats why its so good

true fax

[twilit smelt]

probably

[twilit smelt]

most of these improvements for me feel like i accidentally walked into the women's shower and i just barely managed to escape without being noticed or something

[twilit smelt]

there are JIT libraries that could be useful

i think in this case thatd be like putting a jet engine on a horse drawn carriage though

[acoustic sparrow]

copy and patch doesnt even need a library really

[twilit smelt]

like dude its only supposed to go at like 25mhz anyway. this work isnt even to make it go faster its to make it run cooler on my laptop

it already runs at 25mhz

[acoustic sparrow]

but also adding JIT would add a lot of complexity

[twilit smelt]

itll be fun to be able to crank it up higher though but i usually wont

[twilit smelt]

wazzat

[acoustic sparrow]

basically, you generate chunks of code that you can stitch together into a baseline jit output

to remove dispatch overhead

[twilit smelt]

i think i get it

did i tell you that like 5 years ago i tried to do a "jit" by generating plain text lua code and then string.load'ing it

[acoustic sparrow]

yeah

[twilit smelt]

(in luajit)

i remember specifically it made the memset able to do like 2gb/sec but then everything else was devastatingly slow and it was a little comical

thinking back

it really should have been faster than that

so i probably did something stupid

like yes its slow to parse lua code every time i execute a new block but also its being compiled by a proper, competent, good actual jit (luajit)

and the former should be relatively uncommon

[twilit smelt]

"jet engine on a horse drawn carriage" is one of my favorite analogies

my favorite thing it applies to is the NT kernel underlying Windows

[twilit smelt]

i had a potentially clever idea for dealing with the zero register

previously i would zero it out every cycle

(to avoid a branch on whether the register im storing a result to, is the zero register, or whatever)

the naive initial thing im doing rn is to zero it out in the implementation of each instruction that writes to a register

but later what ill do is

in the decode functions, ill replace any reference to zero as a destination register, to a fake 33rd register

and ill have an extra slot in the register file for that

but zero as a source register will remain just zero

[acoustic sparrow]

there is an even better approach

you can specialize all of your opcode handlers to a given set of the registers being zeros

so you generate ori_arg0notzero_arg1notzero, and ori_arg0notzero_arg1zero, and ori_arg0zero_arg1notzero and ori_arg0zero_arg1zero

and then select the appropriate variant at decode time

[safe hollow]

you only need it for writes though

if there is no write to zero, then no read can be nonzero either

you can do it as a small peephole opt tho, for stuff like addi

[acoustic sparrow]

oh yea that too

i think its better to specialize on reads not writes

because writes to zero dont matter

and arent that common

while reads from zero are very common

[safe hollow]

well you can just remove pure instructions that write to 0

[acoustic sparrow]

that too

[safe hollow]

so this just leaves the special cases like jalr

(no idea what xrarch does different from riscv here, but there should be few)

[acoustic sparrow]

yea its similar to riscv here afaik

[safe hollow]

also why do you need to do LRU with a linked list

when you can just do CLOCK and the likes

which make it just a single write to the basic block info header, or a bitarray

[dense vigil]

hey saga

[safe hollow]

hi

[dense vigil]

when new shitos

[safe hollow]

idk

[dense vigil]

[twilit smelt]

first signs of life

[twilit smelt]

wowie

[safe hollow]

oh also another thing you can do is move some shared work before the dispatch

[twilit smelt]

trying to boot any of the OSes with the new emulator core causes a hang except for aisix which at least panics

[safe hollow]

so instead of doing ```c
void Add(Inst *inst, u32 *reg) {
reg[inst->rd] = reg[inst->rs1] + reg[inst->rs2];
inst[1]->fn(&inst[1], reg);
}

or somethign along these lines, you can do the rs1 and rs2 load before the dispatch
```c
void Add(Inst *inst, int s1, int s2, u32 *reg) {
    reg[inst->rd] = s1 + s2;
    inst[1].fn(&inst[1], reg[inst[1].rs1], reg[inst[1].rs2], reg);
}

[twilit smelt]

they all crash with an unaligned address exception so im guessing i subtly mis-implemented an instruction

[twilit smelt]

this is a more interesting crash

[twilit smelt]

interesting illegal instruction exceptions

[blissful smelt]

is there an AISIX progress thread

this seems new

is it another mintia-level os

[twilit smelt]

idk if youre joking but aisix is from like 5 years ago

[mortal thunder]

AISIX is old as hell

older than mintia

[blissful smelt]

oh im not that old

ok i shall take a gander

[twilit smelt]

it was my first semi-viable kernel project

[blissful smelt]

ohhh i see it on github now

yes cool beans

[twilit smelt]

current state of old mintia on the new emulator core

and new mintia hangs here infinitely faulting

all 3 OSes are very sick in different ways

very scary

[mortal thunder]

why is it caching 295 FCBs

what's that for

[twilit smelt]

its for FCB caching

[mortal thunder]

sure but why are you caching 295 of them

surely no files have been accessed before IOInitPhase1

[twilit smelt]

thats the maximum

[mortal thunder]

ah, okay

[twilit smelt]

its some fraction of the size of physical memory

[mortal thunder]

so you cant have more than that many cached at a time

i see

[twilit smelt]

to be exact, you cant have more than that many with a zero refcount at a time

[mortal thunder]

yeah i figured

[twilit smelt]

theyre only considered to be in the cache when their refcount is zero

otherwise theyre just in use

and cant really be yoinked out

[mortal thunder]

surely you can have more than 295 unique FCBs in the system

is what i thought

and you confirmed

but also isn't the file system supposed to be responsible for FCB caching, or how does this work

oh i guess the filesystem adds FCBs to this cache when their refcount is zero and removes them when they are needed again

[twilit smelt]

basically

it wont work like that in mintia2 tho

[warm pine]

he's not from our generation, his work will have been driven by conscientiousness, not neuroticism

[twilit smelt]

i dunno

[marble socket]

oh yeah i cant remember but what is aisix again

[marble socket]

new emulator core yoo

[night needle]

why is there a / between XR and station

[marble socket]

also that yea lol

[marble socket]

also to be fair, i wanted 1 but i said 2 cause everyone else was saying it, you could say i conform to normality 💀

[night needle]

i make polls to see if people agree with what i want to do regardless

[marble socket]

fair

i usually do with what a poll says unless i dont feel like it or its some joke poll

[twilit smelt]

shitty unix clone i was doing in like summer 2020

before i got into NT

[marble socket]

ahhh right

[twilit smelt]

ive ported it upwards through all the architectural revisions so that i can say i have 3 OSes for my fake computer

[marble socket]

chad behavior

[twilit smelt]

theres also a compatibility bit in the architecture specifically to be able to run it

[marble socket]

thats cool

[twilit smelt]

it causes paging to be turned off when an exception is taken

which is something aisix relies on but neither of the mintias do

(its kernel runs in the physical address space)

[marble socket]

well that seemed retarded

well i mean

[twilit smelt]

yeah at the time i didnt understand how a kernel would run with paging enabled

i was dum

[marble socket]

😭 honestly fair

[twilit smelt]

things you get confused about when youre learning computers by creating one

[marble socket]

Id imagine so

[twilit smelt]

i think its stable enough for me to push

@acoustic sparrow i had a clever idea for minimizing basic block invalidation on tlb flushes

i can have an array that shadows the tlb, of list heads for each basic block that resides within each vpn, and when i flush a vpn i can just walk that list and invalidate all the basic blocks

also behold. https://github.com/xrarch/xremu/blob/main/src/xr17032fast.c

this is an initial pass without doing too many fancy optimizations

it seems to boot and run the OSes fine

lol

seems to

who knows what evil bug could still be lurking

oh i just thought of one

i forgot to re-add privilege checking

xr17032 usermode has godlike powers

for the next like hour

HACK IT HACK IT HACK IT

NOWS YOUR CHANCE

0 DAY 0 DAY

[acoustic sparrow]

yea that could work

i would do it lazily, when you enter the block

where you use the fact that you write TLBs in a specific order

so you can quickly check if you invalidated any TLB entry that may be in the cache

[twilit smelt]

I just realized that'll cause excess invalidations because a simple entry replacement will need the chain of blocks to be invalidated

So I might do some variation upon that instead lol

[acoustic sparrow]

and then you just revalidate it against the itlb

and revalidate the whole inline cache

[twilit smelt]

i think what ill do instead is like

ill just change the hash function to include a few bits from the vpn and then when i invalidate a vpn i can scan that range of buckets for blocks to invalidate

something like that

This is soooo satisfying I can't believe it works and that it's fast

[distant topaz]

ounted

[distant topaz]

This reminds me of OpenBSD

[marble socket]

very epic

[twilit smelt]

if anyone was curious the dragonfruit dhrystone (compiled with an insanely bad compiler) can now do 44-45 VAX MIPS under the cached interpreter (on my m1 mba)

it was capped at 20 before

the emulator is running about 80 million instructions per second

come a long way from the lua emulator that could do like 2-5 mips

lol

also ill repost what i posted in my discord here

also i added i can probably completely remove the dispatch loop and turn its functions (checking for pending interrupts etc) into another tail called routine

[twilit smelt]

sorry i aint never getting back to mintia2 in our lifetimes ive been thoroughly nerd sniped by the emulator problem

[mortal thunder]

and here i was thinking my yapping was excessive

👏

[blissful smelt]

reminds me of some NT driver examples having 700 line functions that are 75% comments and 25% code

great stuff

[tame phoenix]

yes agreed I love it when something is clearly explained instead of just a function with no comments and 3 params called a, b and c

[blissful smelt]

scientist code lore

[icy bridge]

@twilit smelt the xr17032 handbook lists the hex version of the pause function code incorrectly

[icy bridge]

oh and this description contradicts the pseudocode (description says RB is written and RA is ignored, pseudocode says RA is written)

[dense vigil]

did u actually read the entire thing?

[twilit smelt]

You should feel lucky I documented it at all.

Jk I'll fix it

Thx

[icy bridge]

just noticed it

i'm working on porting binutils to xr17032

so i'm looking at this stuff a lot

[dense vigil]

waiiit

are u making a mintia x86 port?

or well

[icy bridge]

well the end goal is getting linux running on xrstation

[dense vigil]

holy

when did u start working on that

[icy bridge]

yesterday

[dense vigil]

thats crazy

why gcc and not llvm?

[icy bridge]

linux userspace depends on gcc libs

[night needle]

peak

[icy bridge]

specifically libgcc_s.so must be present for various stuff to work

[dense vigil]

if you can actually get that done that will be absolutely isnane

[blissful smelt]

wait how fast is the emulator rn

[dense vigil]

slow as fuck

[blissful smelt]

like mhz

oh ok

does linux play well with cpus that run at 50mhz

[night needle]

kvm when

[icy bridge]

current state: binutils compiles, its basic utilities work, objdump when disassembling hits a todo assert, ld is untested, as parses & assembles instructions correctly but doesn't know how to deal with non-constant immediates yet (this is what i'm working on atm)

[dense vigil]

wdym? it will boot just slowly

[blissful smelt]

oh ok

[dense vigil]

like it doesnt care

[teal trench]

you can run it on an amiga or a 68k mac

so yea

[dense vigil]

about the speed of your cpu

[dense vigil]

daamn

[night needle]

how would linux work on xr? dtb?

[dense vigil]

hardcoded platform like m68k on qemu

[icy bridge]

i'll probably provide a dtb when i get there

[night needle]

based

[dense vigil]

but a dtb would be nicer yeah

[icy bridge]

xrstation acpi when

[night needle]

uefi when

[dense vigil]

is it little endian?

[icy bridge]

yes

[dense vigil]

if it is then i see no issue

[night needle]

wait does acpi not work on be?

[dense vigil]

nope

[night needle]

acpi > dt because it doesn't work on BE

[icy bridge]

really? i thought it's just that you have to convert the endianness of all the data

[blissful smelt]

lore

[dense vigil]

on paper yes

[blissful smelt]

so stuff like ppc be just dies?

[dense vigil]

but in reality nothing supports that

[blissful smelt]

rip

[icy bridge]

ah fair

[twilit smelt]

It just got 2x faster yesterday and today I might eke out yet another 2x

[dense vigil]

modern ppc is le

thats cool

did u know about the upcoming linux port?

[twilit smelt]

Well they said they started working on it yesterday

[icy bridge]

very happy with my progress so far ```
$ host-pkgs/binutils/usr/local/bin/xr17032-unknown-linux-gnu-as test.s -o test.o
$ host-pkgs/binutils/usr/local/bin/xr17032-unknown-linux-gnu-objdump -x test.o

test.o: file format elf32-xr17032
test.o
architecture: xr17032, flags 0x00000010:
HAS_SYMS
start address 0x00000000

Sections:
Idx Name Size VMA LMA File off Algn
0 .text 000001f0 00000000 00000000 00000034 20
CONTENTS, ALLOC, LOAD, READONLY, CODE
1 .data 00000000 00000000 00000000 00000224 20
CONTENTS, ALLOC, LOAD, DATA
2 .bss 00000000 00000000 00000000 00000224 2**0
ALLOC
SYMBOL TABLE:
00000000 l d .text 00000000 .text
00000000 l d .data 00000000 .data
00000000 l d .bss 00000000 .bss
00000000 g F .text 000001f0 _start

$ host-pkgs/binutils/usr/local/bin/xr17032-unknown-linux-gnu-readelf -j .text test.o

Hex dump of section '.text':
0x00000000 06000000 07000000 06000000 07000000 ................
0x00000010 06000000 f9060090 0a000000 07000000 ................
0x00000020 290000c0 06000000 3d000000 75000000 ).......=...u...
0x00000030 ad000000 e5000000 15010000 5d010000 ............]...
0x00000040 8d010000 c5010000 3c4a0000 b45a0000 ........<J...Z..
0x00000050 2c6b0000 a47b0000 1c8c0000 949c0000 ,k...{..........
0x00000060 0cad0000 84bd0000 390600f0 3b060000 ........9...;...
0x00000070 3b060000 f3060000 6b070000 f90200b0 ;.......k.......
0x00000080 3a580000 3a580000 32600000 2a600000 :X..:X..2..*..
0x00000090 1a000000 12000000 0a000000 f8620000 .............b..
0x000000a0 f90200f0 f90200f0 f90200f0 f90200f0 ................
0x000000b0 f90200e0 f90200d0 790300b0 790300b0 ........y...y...
0x000000c0 790300b0 790300b0 790300a0 79030090 y...y...y...y...
0x000000d0 f9620d80 f9620d84 f9620d88 f9620d8c .b...b...b...b..
0x000000e0 f9620d70 f9620d78 f9620d6c f9620d50 .b.p.b.x.b.l.b.P
...```

[blissful smelt]

wait are we writing compiler backends for xrstation

neat

[dense vigil]

cool

the xr bootloader probably cant boot an elf

but thats not a problem

[icy bridge]

xr firmware just loads x sectors from offset y at address 0x3000 and jumps to that

i will need to make my own bootloader though yes

[dense vigil]

could just do everything inside linux then

like an efi stub

[icy bridge]

also holy shit gnu as backend interface is horrible, like backend just gets a string for each instruction and everything from parsing it to emitting the bytes for it has to be done manually

void
md_assemble (char *op)
{
  // ...
}``` this is the entire "assemble an instruction" interface

[twilit smelt]

I did my assembler backend by having the general parser break into the backend parser code which can use the common lexer

[dense vigil]

lol

[icy bridge]

op is just the text from the file with whitespace removed

[twilit smelt]

Which sounds better

[dense vigil]

wtf?

what about ir etc?

[icy bridge]

does not exist

[dense vigil]

there isnt an ir in gcc?

[twilit smelt]

https://github.com/xrarch/newsdk/blob/main/XrAsm/xr17032/XrTarget.jkl

[icy bridge]

this is binutils not gcc

like one of the instructions in my test.s is mov long [s14 + s15 rsh (1f - 1b)], s16, guess what gets called? md_assemble ("mov long [s14+s15 rsh(1f-1b)],s16")

[dense vigil]

ah u mean like GAS?

[icy bridge]

yeah

[twilit smelt]

They're talking about gas not gcc

[dense vigil]

lol

at least the whitespace is removed

[icy bridge]

and then md_assemble is responsible for everything from parsing to outputting the bytes and relocations

[twilit smelt]

Surely there are utilities to help you deal with this

[icy bridge]

i took a look at how other arch backends deal with it and nope

every backend just provides its own handrolled instruction parser

which it then calls in md_assemble

[twilit smelt]

My way is way better then

[dense vigil]

when u get paid per loc

[icy bridge]

the only common code is that which is responsible for parsing expressions

like the (1f - 1b) in that example gets handled by common code

but everything else the arch backend is responsible for

my parser is currently 519 lines and that's for a super simple arch

and like thats doable

but for stuff like x86? absolutely horrible

[dense vigil]

is your code available anywhere?

[icy bridge]

only in a private repo for now and when i do make it public it'll be a giant .patch file since i'm using jinx

here's the one for the current version though

[dense vigil]

ngl it should be a repo in the xr org

[dense vigil]

lol u even followed their crappy code style

[icy bridge]

i mean yeah

i might highly dislike it but i'm gonna follow what the project dictates

luckily kate has a setting for the mixed tabs/spaces indent that gnu uses

[dense vigil]

waait

mixed??

wtf does it do exactly

[icy bridge]

2 space indent, but every 8 spaces is replaced with a tab

you get stuff like this

[dense vigil]

wait what the actual fuck

why every 8 spaces

[icy bridge]

because thats what they assume a tab does, align to the next 8 column boundary

[dense vigil]

thats not how you tab

[icy bridge]

so if you're using the same editor settings as the devs and you aren't looking at a patch or something else that prefixes each line it'll look identical to pure spaces

if not? get fucked

[twilit smelt]

ive literally never heard anything good about gnu

everyone ive ever seen who did anything with gnu code was ranting and raving about how awful it is the entire time

[icy bridge]

and i will corroborate that experience

[night needle]

gnulib is fucked
gnu servers are slow
coding style is bad
glibc is full of hacks
autotools

though the llvm coding style also kinda sucks

[mortal thunder]

i seriously hope you manage to finish this

[blissful smelt]

is there any logic behind why these things are so horrendous or is it just for historical reasons

[safe hollow]

common sense hadn't been invented yet back then

[twilit smelt]

porting linux might end up being the easy part

[icy bridge]

i think the two hardest parts are going to be binutils and gcc, by far

opposite order probably, gcc harder than binutils

i think the only two other packages that need any significant amount of arch specific code are linux and glibc

[mortal thunder]

yes

exactly

[icy bridge]

are there any already-made xr17032 disassemblers that i could use to inspect gnu as output with?

or will i need to do manual decoding / compare with xrasm.exe output

[icy bridge]

also in the handbook the pseudocode for the branch instructions is missing the left shift

[twilit smelt]

oops

[twilit smelt]

i just spent an hour debugging why old mintia was suddenly "stuck" here after adding the optimization where i redirect the zero register at decode time

before realizing that last night i did this and forgot

i commented out the video console's login session >:(

there was no bug

[blissful smelt]

what is new xremu clock rate in mhz

[twilit smelt]

its still 20mhz as always

the max you can crank it up to depends on your pc

on my m1 mba it can do about 80

now

previously it could only do like 35

[blissful smelt]

i shall try on my machine, how do i find out the mhz it is running at

[twilit smelt]

run dhrystone at 20mhz and get the vax mips number (should be ~11)

then set the emulator to like 150mhz or something else that youre reasonably confident you cant actually run it at

run dhrystone again and get a second vax mips number

divide the second vax mips number by the first

multiply the result by 20

and that should be a rough estimate

lol

[blissful smelt]

ok i have ./xremu so now what is the next step

hmm ima go find the dhrystone thing

[twilit smelt]

theres a dhrystone in this old mintia image

./graphical.sh -dks mintia-XRstation-fre.img

[blissful smelt]

so

[twilit smelt]

dhrystone

[blissful smelt]

ok yea i got about 11

👍

[twilit smelt]

now run with -cpuhz 100000000 or some other large number (thats 100mhz)

run dhrystone again

report back

@signal zinc i saw you typing.

[blissful smelt]

i set it to 500mhz and it didnt report big number

it booted pretty fast though

on M4 macbook air

[twilit smelt]

youre running it at about 108MHz

[blissful smelt]

i guess that is the max for this system then

coz i did

> ./graphical.sh -dks mintia-XRstation-fre.img -cpuhz 500000000```

[signal zinc]

i think i was pressing space to pause youtube

[blissful smelt]

yea i did 1ghz and it was about the same

unfortuntate

[twilit smelt]

why is that unfortunate

108mhz is crazy fast

[blissful smelt]

big number no go up 🙁

[twilit smelt]

it went up to 108mhz

you would have gotten like 40-50 before the work of the last few days

then 108mhz would have seemed very exciting

lol

[blissful smelt]

ohh ok wowzers thank you mister sky this is legendary performance engineering work you have accomplished

I'll try again on an x64 host tomorrow that supposedly has a clock speed of 4.5GHz but idk if the difference will be noticeable

I saw it at 4.9 once so maybe we will get bigger number

those CPU overclockers that use liquid nitrogen to get up to 9ghz should be running xremu

[twilit smelt]

try it again

with the most recent commit

i wanna see something

[blissful smelt]

👍

number went up

very incredible work

[twilit smelt]

went up by 30%

that was a like 20 line change btw

[blissful smelt]

removed dcache locks 👍

[twilit smelt]

no they are still there I just avoid locking them in the hit case

[blissful smelt]

ohh ok neat

next up xremu rewrite in assembly

[icy bridge]

update on the xr17032 linux port: the binutils suite seems to be fully functional, tomorrow i will test it a bit more and maybe start working on gcc ```
$ host-pkgs/binutils/usr/local/bin/xr17032-unknown-linux-gnu-as test.s -o test.o
$ host-pkgs/binutils/usr/local/bin/xr17032-unknown-linux-gnu-ld -o test test.o
$ host-pkgs/binutils/usr/local/bin/xr17032-unknown-linux-gnu-objdump -d test | head -n 15

test: file format elf32-xr17032

Disassembly of section .text:

08000054 <_start>:
8000054: 96 0a 00 00 j 548 <_start-0x7fffb0c>
8000058: ae 00 00 10 j 8000054 <_start>
800005c: af 00 00 10 jal 8000054 <_start>
8000060: 3e 01 00 10 j 800009c <_start+0x48>
8000064: 3f 01 00 10 jal 800009c <_start+0x48>
8000068: de 00 00 10 j 800006c <_start+0x18>
800006c: f9 ce 1a 97 mov long [s14 + s15 RSH 24],s16
8000070: 4a 26 06 00 mov long [s14 + 0x18],0x4
$ host-pkgs/binutils/usr/local/bin/xr17032-unknown-linux-gnu-readelf -h test
ELF Header:
Magic: 7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00
Class: ELF32
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: EXEC (Executable file)
Machine: XR/17032
Version: 0x1
Entry point address: 0x8000054
Start of program headers: 52 (bytes into file)
Start of section headers: 740 (bytes into file)
Flags: 0x0
Size of this header: 52 (bytes)
Size of program headers: 32 (bytes)
Number of program headers: 1
Size of section headers: 40 (bytes)
Number of section headers: 5
Section header string table index: 4```

[teal trench]

nice

[dense vigil]

[twilit smelt]

are you serious how did you do that that fast

Even the linker??

[icy bridge]

the linker was actually incredibly easy

I only had to add like 15 loc in the ld directory, ld.bfd is incredibly generic

[dense vigil]

why does it do random jumps?

[icy bridge]

test.s just has an instance of every form of every opcode

[twilit smelt]

Even for the relocation types and stuff?

[icy bridge]

it's not actual code that does something

[icy bridge]

yeah that's all handled by bfd

[dense vigil]

ohhh

[icy bridge]

this is the entire binutils patch https://github.com/monkuous/xrlinux/blob/main/patches/binutils/jinx-working-patch.patch

3000 loc for an entire new architecture, not too bad

ofc ill probably have to expand it for stuff like cfi, pic, etc. but all the core functionality is there

[twilit smelt]

Just in time for the emulator to be fasta too

[icy bridge]

yeah I was curious how fast it could get on my machine and according to the procedure you mentioned yesterday I'm reaching like 170 MHz

[twilit smelt]

It's about 6x improved from before and can now do >200 real MIPS on my m1 macbook air

[icy bridge]

~105 vax mips

[twilit smelt]

130 vax mips is the highest I've seen it go

If you comment out smp support stuff it can go 160 but that's cheating

[dense vigil]

maybe you should have a relaxed mode that doesnt do cache/latency emulation

[twilit smelt]

Yeah I was thinking I might conditionalize that stuff out for special (lame) builds

You could still do smp but they'd all be multiplexed on a single host thread

[icy bridge]

first binutils-based xr17032 output

[dense vigil]

like qemu with tcg doesnt try to be that accurate with this stuff

[dense vigil]

damn

[icy bridge]

this is what produced that disk image https://hst.sh/ajusicetar.yaml ```
$ cat build.sh
#!/bin/sh
set -ue

host-pkgs/binutils/usr/local/bin/xr17032-unknown-linux-gnu-as test.s -o test.o
host-pkgs/binutils/usr/local/bin/xr17032-unknown-linux-gnu-ld test.o -o test -Ttest.lds
host-pkgs/binutils/usr/local/bin/xr17032-unknown-linux-gnu-objcopy test test.bin -Obinary
dd of=test.bin bs=512 oseek=512 count=0 status=none```

[dense vigil]

also if you add a simple jit for your emulator that would probably give another order of magnitude of speed

[twilit smelt]

Emolator

[dense vigil]

@icy bridge btw how hard do you think would it be to add a QEMU target for this + tcg?

[icy bridge]

no idea

it's a pretty simple arch but idk how complex a basic qemu backend is

[dense vigil]

yeah not sure

[icy bridge]

also @twilit smelt i think i found an a4x bug: if AptOsRecord.BootstrapSector+AptOsRecord.BootstrapCount == AptEntry.SectorCount booting fails with Failed to read bootstrap program

here's a reproducer disk image

specifically i'm pretty sure https://github.com/xrarch/a4x/blob/df03055a971767d16140be26b1e12d9f0e0cea64/a4x/Fw/FwDks.jkl#L114 should be > instead of >=

[twilit smelt]

Inchresting

5 year old project gets a second set of eyes on it for the first time and 500 problems are immediately identified

[mortal thunder]

(this code is not even 2 years old)

[twilit smelt]

So

It's part of a 5 year old project

7 years technically!