Keyronex

[zenith karma]

but not sure, i use it as a cheap (for me) way to get rid of an fs subtree

[hexed solstice]

i think i might actually go ahead at once and RCU the mount table - i simply cba implementing it with locking when i have already got a non locking solution

i'll need an RCU'able hash table

[hexed solstice]

my buffer cache doesn't retain vnodes for which a mapping exists in the cache - i wonder if i want to do so - i'm trying to remember what my rationale was for not doing so

[hexed solstice]

i really don't know. it doesn't make good sense to me why i did this

i think before i might have conflated "references to vnodes" with "reasons to prevent an unmount operation"

but now i know that this is a coarse approach and a better one is to explicitly count open files instead of counting the vnodes

the vnodes are still refcounted and would have to be gotten rid of befoer proceeding with an unmount, and that must include scanning them and writing their dirty pages back

but that can be part of the unmount operation

[twin haven]

The feeling when coming back to your old code

[glad sentinel]

For me at least the answer is usually some combo of 1) college code and 2) 4am code

[hexed solstice]

i even have a comment block in which i describe how the cache can work without retaining a reference tot he vnodes whose contents it caches

and i explain to the reader (myself) how it is legal

but not why i have done this

and my feeling right now is that this is wrong

wrong as in it's wrong not to reference them because, if it did reference them, that's more natural, and it provides a form of vnode caching beyond what the namecache does

right now anyway i am carefully working out namecache subtree destruction

i need to watch the refcounts closely, make sure it interacts properly with the namecache LRU mechhanism, and ideally do it in an iterative not recursive fashion (because who knows how deep might a hierarchy be?)

this is the first part of unmount support. the second part is buffer cache purging. the third part is the open files count thing for mountpoints which we worked out two nights ago. the fourth part, that one i haven't gotten to yet

[hexed solstice]

oh no

the code i wrote in 2023 is missing essential features

wait never mind

[hexed solstice]

i don't like the LRU logic in this at all, it looks fragile

maybe i thought through all the problems when i first wrote it 15 months ago, and i have certainly been able to achieve something, but i remain sceptical anyway

and now i would probably want to use RCU somewhere

[hexed solstice]

the first use of RCU in keyronex is arriving: freeing of VFSes

that's going to be deferred and traversal of mountpoints is going to go through an RCU-safe hash table

[hexed solstice]

unmount progress:
💯 namecache purge seems to work
💯 UBC purge seems to work
❌ dirty page writeback awaits to be done

so why not now implement fsync and the sync op on a vfs (i.e. fsync everything + metadata.) the right thing to do is probably to introduce a per-VFS queue of vnodes and use that for this; a lot of FSes do naturally have a structure containing all vnodes (anything unix-like as far as inode vs dirent distinction the fs driver has an rb tree mapping inode number to in-core fs node) but that's not ideal for this (because the release vnode op usually wants to acquire that lock, so you'd have to defer invoking that, but vnodes are supposed to be released quite as soon as they each 0 dirty pages)

[hexed solstice]

that's an "all vnodes of this vfs" list now added to the vfs struct, and APIs for iterating all vnodes of some vfs

watching refcounts doing what they are supposed to do is satisfying

no fsync yet - that is still to be implemented

[hexed solstice]

hmm

i was thinking of adding a size field to vm objects and strictly faulting on higher than the size

but i am not sure if that's the best way to do things for 9p and nfs

maybe it's nicer for mmap() on those to try it's best and see whether it can take in some data

and the same may go for read

[hexed solstice]

i am now inclined to do this - i think with network FSes one has to make certain compromises

i can always have it behave differently for the default read/write ops

i have added also a vm object to map entry list - every VM object gets one - the purpose is to allow for the semantics posix prescribes for file truncation, to wit, that after truncation, future acces to pages beyond the truncation point generates SIGBUS

i am not going to do the crazy linux behaviour, where truncating a file causes any private anonymous pages beyond the truncation point (i.e. generated by write faults on a MAP_PRIVATE mapping) to be chased out too. i think that behaviour is strange and surprising

https://github.com/Keyronex/Keyronex/blob/619ef9e3c819b8243e17e30c0d007af85eff1491/kernel/vm/map.c#L2-L28 and i took the time to write a brief comment on how the new vm boject to map entry mapping fits in

[hexed solstice]

i threw together a quick diagram of how namecaches are working in keyronex now

this shows how a nullfs (on linux called a "bind" mount), which mounts / over /mnt, works

[twin haven]

mm I'm saving this one, it'll prove useful in the future I think

[hexed solstice]

who is following a link from serenity os to keyronex repo?

@stone orbit i think i've found what stopped your build

i only make a crt0, not a crt1

[stone orbit]

would that crash the compiler though?

[hexed solstice]

no, that one i can do nothing about

[stone orbit]

oh :(

[hexed solstice]

linux distros often do very bizarre things with compiling

using extreme optimisation flags, and they also strip everything for some raatid reason

[sinful jetty]

How's that even possible

[final hatch]

the more important question is

How are they tracking that shit

[stone orbit]

probably https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Referer

but i wouldn't know

i'm not enough of a web dev to know stuff like that

[final hatch]

i see

[stone orbit]

use jinx instead

or xbstrap + cbuildrt, but that requires you to prepare a container image

and i believe it is opt-in as opposed to forced containerization like jinx, so that's up to you if you do decide to support some sort of reproducible builds

[zenith karma]

you can strip to external file?

i think arch strips binaries to external

and then they publish -debug packages

(which also contain sources, so now as part of my system software update i download a new copy of all of firefox and glibc :^))

and arm gcc for some reason

[hexed solstice]

not everything can deal in those externalised symbols

and they do this really hardcore extreme strip there

it's not as though they're afraid of using loads of disk space on linux (just look at Snap! and FlatPack)

[zenith karma]

really?

what can't?

[sinful agate]

It is. That said, highly recommended

[hexed solstice]

going to do shared anonymous VM objects next. then possibly fork, or maybe i'll leave that for later. with that, i think i have enough groundwork laid to invent an IPC mechanism and then begin the POSIX server

as it happens also last night i had a dream where i was instructed that a fun way to make page state locking fine-grained is to lock the table from which a page is referenced

with the lock being a single bit of the flags field in the vm_page_t structure; you can do a compare-exchange loop to acquire and to release it

[final hatch]

How are you going to set only one bit

[hexed solstice]

cas

[final hatch]

CAS works on booleans though doesn't it

or wait no

compare and swap, wouldn't that work with integer types?

and not with a single bit

[hexed solstice]

yes, i don't think many architectures have cas on a single bit

that would be quite something

talk about bit-addressable memory

[final hatch]

Hmm so I suppose you're going to CAS the PTE to just the locked flag, manipulate the PTE value and then place it back in its slot?

[hexed solstice]

i was actually thinking of just CAS'ing a flags field in the vm_page_t structure

but i wouldn't do that

i would CAS so that only the lock bit is set

[final hatch]

Oh wait you're talking about vm_page_t and not a PTE

[final hatch]

Ditto but substitute PTE with vm_page_t flags

[hexed solstice]

i would only be setting the locked bit and not changing any of the other flags

[final hatch]

I don't know how you're going to pull that off with CAS

Afaik you can't perform an atomic OR fetch

like you can an ADD fetch for instance

also what would the comparison condition be anyways

[hexed solstice]

while true do
  value = atomic load(page->flags);
  r = atomic cas (page->flags, expected: value, desired: value | locked);
  if (r) break;
done

[final hatch]

oh, i see

looks quite good

better than my thing

[hexed solstice]

what's your thing?

[final hatch]

The one I said above

Where you replace the flags value with just the locked bit, mess with the flags and then replace them back to unlock

like

while ((flags = Exchange(page->flags, FLAG_LOCKED)) & FLAG_LOCKED);
//...
Store(page->flags, flags);

[hexed solstice]

oh yeah, that one might not work always

i'm not sure actually, i am thinking very ahead here

i would probably want to do a simpler approach to locking than this before i went to these radical lengths

at present i have a very big spinlock that protects the state of all trans PTEs

i have a plan of 4 stages to break that up

1. curtail use of that spinlock to only the places where it's absolutely necessary
2. split it into two locks: a pte state lock and a page queues lock
3. split the pte state lock into a lock per owner (vm object or process)
4. split the lock per owner into per-page-table locks

but i will only move through these stages as and when i feel that scalability with the previous solution is unsatisfactory and the problem is contention around the locks

[final hatch]

oh this could be used to set or clear a bit atomically-ish, in any situation

that's smart

[hexed solstice]

yea, or several

it's called the CAS Loop Pattern

very useful pattern

[hexed solstice]

someone in the #osdev irc on freenode raised an issue

because i am using 4-level tree to keep track of pages belonging to file caches, i have about 20 kib of overhead for every open file that I/O is ever done to/from, and no matter whether the file is only 50 bytes long

it's not unfixable, the right answer is probably to separate the handling of anonymous VM objects and vnode-backed VM objects. for anonymous VM objects i all-but-require the tables to be page-sized so that they can be pageable. but this does leave a stiff penalty for files on a tmpfs! every file with even a byte of content on a tmpfs will use up 20kib. not nice

so a variable-depth radix tree might be the best option for the long term, keeping tables page-sized if possible because that works better

an interim solution which might actually be adequate:

• let vnodes keep a small buffer, maximum of perhaps a page in length, which is used as a small cache for their data
• upgrade this to a full VM object only when the vnode is memory mapped or when I/O is done outwith the first page

(use case example: you are using GNU MAKE and GCC and there are a multitude of .d depfiles, each with not much data in them)

[hexed solstice]

in the meantime i have implemented what i now think is most of the work necessary for shared anonymous VM objects, and i also wasted time redesigning the README and re-enabling weekly-build workflows, now for both m68k and amd64

perhaps (for the sake of @stone orbit) i can figure out this feature that xboxstrap is said to have, where you can package up tools and make them available to others

[stone orbit]

what i was talking about is "cbuildrt" which is xbstrap running in a container, which technically should make keyronex builds reproducible given that people also build it the same way

one downside of that is you need a rootfs for the container (like debian or whatever)

you could use managarm's cbuildrt rootfs i suppose, but you're gonna get a bunch of useless packages that the cross compiled packages require at build time (nasty)

preparing the rootfs is probably not the big problem, but you need a way to redistribute it to people who want to build keyronex, at which point it becomes a problem if you don't have a server you can host it on (i suppose you could use gha for that)

also there is something for redistributing built tools, but i have no idea how this works at all, i wont be able to help but it seems to rely on some server to provide these tools

it all seems pretty much tailored for managarm, as expected

[hexed solstice]

i could find a host i'm sure

i will look into it in due time

in the meantime it looks like both amd64 and m68k built fine on ubuntu with gha

i think the next thing i'll do is implement the vm logic for fork

then i might seriously consider inventing an IPC mechanism and starting on the posix server

however for the next 2-3 months and the foreseeable future i am going to be quite a bit busier and won't be able to work nearly as much on this

[sinful agate]

Not really. It’s tailored for xbstrap. It’s “just” a matter of using xbbs (the build server we’ve written for this thing) and using cbuildrt. One should be able to just copy the setup Managarm has (which outside of names and the repo’s it watches to trigger builds is generic) and call it a day. Regarding the rootfs, take Managarm’s GHA job, edit the Dockerfile as needed, and done. Packages will be made as xbps files (from void Linux) if requested. If a different format is preferred, xbstrap and xbbs will accept pull requests to get that added, I’ll personally ensure that

[hexed solstice]

i have thought about this and i think this is the appropriate long-term solution

what isn't 100% clear is whether i could get away with the simple approach of simply allocating a new level whenever we need it, i.e. when a page > 2mib is brought in, go to 2 levels, when > 1gib, 3 levels, etc. it isn't good for very sparse file access

nor for people accessing a few pages located around the ~2gib mark

[hexed solstice]

i might start spamming RCU everywhere - the usefulness is growing more and more

[twin haven]

I was thinking about an idea like this myself some time ago, I'm not entirely happy with how my vm backends represent their objects (and I'd like to unify this actually, right now each 'type' of virtual memory does it differently and its a dance between the vm/backends). I was thinking of some sort of list of trees, so you could represent a memory object with sparse but clustered accesses effieciently, but also the reverse. It just complicates lookup a little bit.

[hexed solstice]

it's all very debatable, i think there is a lot of merit to structures facilitating sparseness

[hexed solstice]

my hope i suppose is that where there is sparseness, there will usually only be a very few pages, enough that i can have (for example) 4 slots in the vm object for pointers to the pages; and when there's more than that, that they aren't ridiculously sparsely spread

[hexed solstice]

right now i am dealing with process and thread cleanup and reference counting correctness around these (which is simple really: once a thread is created it acquires a reference on its process, which persists until the thread is terminated; something that only the thread itself can carry out, so you'd have to send an AST to the thread to get it to kill itself) and once that's out of the way i think i will try to finally design the IPC mechanism - something i really have really little pictuer of yet

there are too many designs that i respect

synchronous L4/QNX/Shuttles for Utah Mach style has merits (and does not prescribe actually-synchronous interfaces! you can have the server call back to the program when it completes an operation), but raises questions, like how servers should be threaded (one thread per connection, for example, and then you might want one connection per client program thread! it is a bit costly); so does asynchronous Mach/Fuhcisa/Managrm style have its own merits

i have usually seen synchronous replacement asynchonous but apparently windows' LPC (though granted i think it's heavier weight than L4/qnx stuff, and doesn't do thread migration like mach shuttles or solaris/spring doors) was replaced with an asynchronous replacement - which is entirely undocumented

[surreal granite]

A(dvanced)LPC supports synchronous model as well, but it was created as a solution to the problems associated with blocking LPC

One of them - inherent deadlocks and the growing code complexity around it

Scalability with LPC was poor

ALPC supports several different models of asynchronous operation. Ports can be configured to perform operation via IOCP instead of the typical request/reply. The port object can automatically balance the number of messages and threads

[hexed solstice]

i have started implementing event ports a la solaris and windows

i have some vaguely formed ideas on how best to scale it

i think kqueue and epoll both suffer a bit in terms of implementation complexity from acting as both event queues as well as pollalikes

both kqueue and epoll are excellent when there is one thread, if you have multiple threads and want a common event queue, they need the EPOLLET/EPOLLONESHOT and EV_DISPATCH/EV_CLEAR flags to not wake every thread, and EPOLLET in particular famously confuses people

[hexed solstice]

it's time to do the object handle table. lockless resolve handle number -> object pointer with RCU of course

and i might just do similar to linux, which has a flat FD array which is reallocated on expansion. it is a simple appraoch

[sinful jetty]

Makes sense because its usually limited to a tiny number by default

Like 1024

[hexed solstice]

it can fairly grow though

that being said i would be shocked to see more than about 20,000 or so FDs open

[sinful jetty]

I could only reach that for a large vm that opened various /dev stuff like vhost-net etc

[hexed solstice]

i am thinking e.g. a web server

[sinful jetty]

Probably

[hexed solstice]

for that and things of a similar class, having more would be excusable, for most other things, less so

[hexed solstice]

i'll just do a flat array too

or rather three flat arrays (one of pointers to open object descriptions, one of cloexec bits, one of used bits - the used bits let you reserve an FD for later use without immediately giving it a pointer value)

as there is a rule that the first free FD is allocated, so the FDs generally don't get too sparse

[stone orbit]

linking chromium is fun :^)

[hexed solstice]

there it is then, the first full idiomatic use of RCU in keyronex

object descriptors are called rights for now because i like the term

[sinful jetty]

Why

What does that mean?

[hexed solstice]

it's a traditionally used term

this is why the mechanism for sending file descriptors over a unix domain socket is called SCM_RIGHTS

the term is also used in mach which is an influence

[sinful jetty]

Does right have some additional meaning I'm not aware of

[sinful jetty]

Thats actually interesting I was wondering why it was called that

[hexed solstice]

it is interchangeable with "capability" basically

[sinful jetty]

But an open file descriptor doesn't really fit that paradigm imo

[hexed solstice]

https://www.cl.cam.ac.uk/research/security/capsicum/

capsicum implements the complete object capability security model with just FDs

they added some options for restricting what can be done with a particular FD in the future and made proceses represented by FDs

[sinful jetty]

Interesting

Will keyronex do that as well

[hexed solstice]

i think i will eventually

capsicum is nice, elegant, it is easy to implement and even easier to reason about

but because it implements a formal security model it needs programs written to understand it

[sinful jetty]

Is it opt in?

[hexed solstice]

it is, someone did implement an alternative ABI where it isn't

that was called CloudABI

it's most notable for becoming the basis of the WASI APIs for WASM - but some idiots misunderstood it and invented WASIX (which not only breaks the capability model but also adds back in fork and signals - features explicitly excluded from WASI for obvious reasons, a new interface which is supposed to improve on unix shouldn't reproduce unix in its entirety!)

[sinful jetty]

lol

[final hatch]

Is this WASM the thing I'm thinking of? (Webassembly)

[hexed solstice]

exactly that, it's a bit of a fad the idea that we should take all our software and run it under webasm runtimes instead

[vale pelican]

how so?

a file descriptor is an abstract handle that grants you access to some resource (file, stuff like eventfd/signalfd/timerfd, etc)

[sinful jetty]

Capability to do what

How do you even word it

[prime coyote]

will you implement more rcu stuff?

[sinful jetty]

Capability to be able to read foo.txt?

[prime coyote]

like sleepable one

[hexed solstice]

that one is patented, i'm not sure if it's expired yet

maybe not because microsoft apparently implemented it into windows

or maybe they independently reinvented a way of doing it, or they think the patent is guff and their lawyers would make mincemeat of it

i have no use yet for sleeping RCU but i think it would be useful to have

[vale pelican]

for example yes

[prime coyote]

what for do they use rcu

[hexed solstice]

omwinyourhead said they use it in the cache manager somewhere

[prime coyote]

interesting

[hexed solstice]

with sleeping rcu i am less keen because there is no call_srcu, that was felt to be too dangerous to implement because of the potentially lengthy grace periods in sleeping rcu, so a lot of operations you've got to synchronously wait for potentially a while. so it is very expensive for a writer

[prime coyote]

wonder if it will be available and documented api

[hexed solstice]

as it happens though i am upset that i won't be able to use classic rcu in the posix server because classic rcu relies on something that user-level can't do (and couldn't do without the expense of syscalls and a lot of trust: inhibiting preemption)

[hexed solstice]

back and working on the handle table again. it occurs to me that i am hampering things a bit by providing the first-free behaviour for allocation

i already considered the possibility of having the object handle table serve as the FD table for posix programs too, but i don't think it would work - i don't believe i can have a 1:1 correspondence such that every posix FD is backed by exactly one object handle, with the objects also carrying enough state to satisfy everything posixy that has to be done with them

so it follows that there will have to be a separate posix fd table and that the necessity to have first-free semantics for native object handle allocation is gone

but i don't need to get ahead of myself. read access (looking up object given a handle number) is already lockless and only needs an IPL raise and two atomic loads under that to get what is almost always the right object pointer. should be fast enough for anyone

[twin haven]

thats very cool

one day I'll have to spend some time and look at your rcu implementation and its uses.

[hexed solstice]

https://github.com/Keyronex/Keyronex/blob/master/kernel/kern/rcu.c it's dead simple thankfully

and as an aid to study interspersed with text from the original outline of RCU

[twin haven]

wow thats straight-forward

[prime coyote]

it's a lot shorter than I thought

what are the quiescent check functions for in rcu? is it part of classic rcu?

[hexed solstice]

these are called at quiescent points

e.g. context switch, timer tick when interrupted IPL < DPC level (in keyronex, or in linux terms, timer ticks when preempt-disable is off)

[prime coyote]

is this another branch of rcu like the sleepable one ?

i saw these in tree.c file in linux rcu

[hexed solstice]

it's the original one

[prime coyote]

on windows i saw KiRcuCheckQuiescent in a call stack for kewaitformultiple function

[hexed solstice]

i think linux abolished RCU classic

RCU Tree replaced it there as far as i am aware

RCU Tree is similar but instead of the global cpu mask there is a tree structure, where you only need to lock your leaf of the tree (shared with, say, 64 other cores) to signal a core quiescing, and when every core has quiesced, then you can acquire the lock of the next level and indicate queiscence there, and so on until you reach the root

[prime coyote]

is it patented?

[hexed solstice]

i think so

[prime coyote]

and there is also rcu preempt

and they're all related to each other?

[hexed solstice]

all descendents of RCU classic

the preemptible variants of RCU are more deviant from the classic though

[hexed solstice]

i have come up with a ridiculous scheme to make it so that a spinlock serialises updates to entries in the handle table against the initiation of resizes

while a (sleeping) mutex serialises the allocation and freeing of handle numbers

this means that spinlock doesn't need to be held during an O(n) operation

[hexed solstice]

(as this probably does not make sense without this context: i have allocation/deallocation of handle numbers be a separate thing from setting them. the original reason for this is because i didn't want to do work to build up an object only for it to be torn down because there was no space to allocate a handle for it. i think i had some other reasons too - i've forgotten them. but that's it's purpose together with letting allocation/freeing take mutexes being O(n) operations right now while installing an object pointer into a slot allocated earlier is not.)

[hexed solstice]

the new RCU handle table is in use now. some refcount correctness still needing to be added in

the object manager includes support for a special try-to-release-final-refcount case - it's invoked when a refcount drop would be made from 1 to 0, and it defers that drop to a per-object-type function. i don't believe i have a use for that anymore with RCU

the main use for it before was to support keeping objects in data structures for cached lookup but where the cached lookup data structure should not retain a reference to the object. well, now i can defer the freeing of the object, and i do support try-to-retain-object-by-pointer, which tries to retain the object only if refcount != 0 (if refcount = 0, object is to be destroyed but freeing has been delayed by RCU.) so i don't think i need that facility anymore

i am keen to figure out how freebsd's global unbounded sequences (gus) works, which apparently has nice properties that RCU doesn't have (but you can't do arbitrary stuff with it, it's strictly for deferring freeing)

[zenith karma]

do you have a generic cross-core message-passing primitive?

it seems to be a nice primitive for building many lock-free data structures

[hexed solstice]

not yet

netbsd has one called xcall and has a mechanism called pserialize based on it

it works like this

[zenith karma]

i think it's a really nice primitive to build off of

(not that i have but you know)

[hexed solstice]

your reader raises IPL on a core to some value you've decided on; you read your data structure

your writer does what it will to the data structure but doesn't delete anything yet; it sends an xcall to every core at this same IPL; it waits for all cores to acknowledge receipt; when all cores acknowledge receipt, the writer can now delete the old things

[zenith karma]

i mean yeah

[zenith karma]

you don't have to wait for a receipt though?

you can just do it asynchronously

[hexed solstice]

speaking in terms of the straightforward synchronous code that could as easily be done asynchronously

[zenith karma]

ah, yeah

[zenith karma]

you can also do something like this (pseudocode): ```py
new_meta = Meta(version: old_meta.version + 1, refcount: len(all_cpus), data: new_value);
for (auto& cpu : all_cpus) {
cpu.run_at_irql(IRQL::ReadCritical, [cpu, new_meta] {
if (percpu[cpu].version > new_meta.version) {
new_meta.refcount -= 1;
return;
}
percpu[cpu].refcount -= 1;
percpu[cpu] = new_meta;
});
}

i.e. async updates (and refcount)

this only allocates once per core and sends half the traffic of an acknowledgement

[hexed solstice]

amazing how much every approach to safe memory reclamation rhymes

you will love XNU SMR (which is similar to FreeBSD GUS) when you look at it

and that's the deferred refcount drop trick dropped from the object manager - from now on i solve this problem with RCU instead

[zenith karma]

yeah, i mean, i don't think there's any other way to do it

or any meaningfully different way anyway

[prime coyote]

wdym

[hexed solstice]

i used to use this trick a lot. let me give the example of a conventional filesystem, one where there is an inode/dirent distinction and where inodes can be uniquely identified by some number

[zenith karma]

you should implement some sort of generic "run this on another cpu" interface over which all this sort of signalling goes over

so that if you have a fancier signalling mechanism that is faster you don't have to rewrite everything

[hexed solstice]

then you want to have a lookup cache of inode numbers -> vnodes, but you don't necessarily want to have that cache retain a reference to the vnode - it's just a lookup cache you need to avoid aliasing an inode with two vnodes, it's not part of the actual filesystem caching mechanics

so when you are releasing a reference to a vnode, what do you do? if you let the reference count go to 0 and free the vnode, then people get confused when they find your vnode in the cache

so you could

lock(inode number to vnode cache);
vnode->refcnt--:
if (vnode->refcnt == 0) {
  remove vnode from cache();
  free(vnode);
}
unlock(inode number to vnode cache);

but that's pure poison to performance

so what you can instead do is to lower the vnode refcnt only if you are lowering it from x to x - 1 where x is any number greater than 1, and if you are lowering it from 1, then don't lower it, call an FS-specific function, which will lock the inode number to vnode cache, then check if the vnode refcnt is still 1; if so, it can remove the vnode from the cache and then free the vnode

otherwise, it can simply return some value that causes the release to be reattempted

an elegant solution which mostly avoids locking the cache unnecessarily

[zenith karma]

this sort of approach is also generic to all lockfree hashtables, isn't it?

[hexed solstice]

an RCU-based cache with the dead vnode preserved long enough to let people see that its refcnt is 0 (= it can't be retained anymore) is even nicer

[hexed solstice]

a truly deferred free is necessary to have lockless lookup, otherwise you need an rwlock to protect your ability to take a reference on something in the cache

[zenith karma]

well, yeah

actually no you don't

maybe

hm

no you do yeah im stupid

[hexed solstice]

and this deferred final refcount release from 1 becomes unnecessary with RCU

since you can just immediately refcnt-- (atomically of course) and then you can unlink it from the lookup cache and put it to RCU or whatever safe memory reclamation mechanism you use. you also need a fallible retain function that will reject a refcnt++ from 0 to 1

[hexed solstice]

you will enjoy this btw https://github.com/concurrencykit/ck

freebsd uses it

[zenith karma]

oh, cool, it's a separate repository even

[hexed solstice]

ck_epoch is the main mechanism for deferring frees. freebsd's is slightly different being a kernel

specifically on freebsd core-based is available as well as thread-based

[hexed solstice]

posix is irking me

i see no other option than to serialise read/write/lseek calls on the same underlying file object against each other

if not, then how do you preserve the atomicity of the read and write ops, where posix says you've got to adjust the offset of the file as part of the same operation as doing the read/write?

[neat steppe]

Maybe you can do an atomic add to the stored offset before starting the operation (including doing whatever file extension is necessary)

And that can be done under a per file lock

[hexed solstice]

that is to say you need to have read look like this:

lock(file->offset_lock);
bytes = read();
file->offset += bytes;
unlock(file->offset_lock);

[neat steppe]

But you release it when you do the actual operation

[hexed solstice]

the trouble is that it's legitimate to pass, say, UINT32_MAX to the count of bytes to read, and let the FS driver or whatever non-FS file you are working with truncate the read accordingly

[neat steppe]

Oh yeah I ran into that same problem

Capture a file size state under that lock which you then pass to the fs call

[hexed solstice]

if your FS driver can give a correct answer to "how large of a read will you give me?" then you can at least just lock around that and not around the whole read - i think

[hexed solstice]

yes, exactly that

i am trying to think of anything that breaks this approach

i can't immediately think of anything other than a network FS TOCTTOU

and of course the contention is only when you have a same file object which several threads (maybe forked off subchildren) trying to access the FD in common to do traditional read/write (as opposed to pread/pwrite) - and that is frankly a really stupid situation to put yourself in

so i don't think i'm particularly upset about this actually - what i do need is turnstiles or pushlocks so i don't have to put a heavyweight mutex to bulk up the file object

[valid sequoia]

aside from the offset issue, they have to be serialized otherwise if you will read stale data

[hexed solstice]

does posix specify that for write? i thought it simply said that if you read after a write has completed, you've got to read only the new data

[valid sequoia]

hmm

[hexed solstice]

i need to redo the scheduler

it's time for per CPU ready queues and possibly dfbsd flavoured

[hexed solstice]

if i don't go to extreme measures with localising to CPUs then i think i need a synchronisation mechanism which is acquired as a thread is going to sleep - so that anyone who should try to wake that thread must acquire this lock to ensure they can't wake it till it's ready

so in that case current thread lock then (if i choose to have a lock around the runqueues and eschew going full DFBSD) runqueue lock

i don't see an immediate need to acquire the next thread lock

once it makes it onto a runqueue for some core, the disposition of that thread scheduling-wise is the responsibility of the local scheduler of that core

if you want to intervene, then either a) less DFBSD style, acquire that core's runqueues lock, and check if thread is still on the runqueue; if not, react accordingly, or b) DFBSD style, send an IPI to that core asking it to do whatever needful thing you want done to that thread

(as it happens this is formalised on solaris, netbsd, i think also freebsd, where a thread has an atomically updated lock pointer which can point to a per-thread lock, a runqueue lock, a wait queue lock, all sorts of things...)

[final hatch]

You should have per CPU ready queues IMO

they're convenient and cool

the problem with how boron implements them is the central dispatcher lock. But if you don't have such a lock you're going to gain a lot from not having to contend on the thread queue lock

[hexed solstice]

you could uncouple the ready queues from the dispatcher lock without breaking the dispatcher lock generally

[final hatch]

I think I could probably do that, yeah, they'd have to have a lock of their own though in the Unwait function which puts a thread to be woken up back on the ready queue

Thinking about it, I don't see why I'd need to hold the dispatcher lock for that

[hexed solstice]

possibly to avoid certain races

that is, you want to signal a thread that's going to sleep, you need to make sure the thread is actually completed entering sleep state before you wake it, lest you have it running in two places at once

so holding a dispatcher lock across the process of going to sleep does the job

[final hatch]

oh yeah

[hexed solstice]

and also when waking it holding that lock so you can serialise the "end wait" step for a given thread - but for the "make thread runnable" step that msut follow, you'd be using probably a per-thread lock there

[final hatch]

Hmm I'd hold the dispatcher lock while setting up the wait (but unlock it before calling yield), and when unwaiting the thread in question

I could make it so that, in that case, before unlocking the dispatcher lock I lock a per thread lock, which is unlocked only after switching context

with some kind of "previous thread" field in the PRCB mayhaps

and then the unwait function would hold both the dispatcher and the perthread lock

[hexed solstice]

exactly that, and i think that could be a nice step for when you get an AMD® ThreadMutilator2000™ in 2037 with 128 cores, and decide you want to start relieving contention around scheduling

[final hatch]

AyyMD already makes such monsters I'm pretty sure

[hexed solstice]

possibly

i forgot if they are doomed or not

because originally intel was doomed, then amd was doomed, then intel was doomed, then amd again, now i think it's intel that are currently doomed

[final hatch]

No I'm pretty sure they're both doing well

[hexed solstice]

i've made an initial go at it but i've made some stupid mistake somewhere i'm sure

IPL staying at DPC level for whatever reason

oh i see it i think

yes, stupid mistake, previously i acquired the scheduler lock around calls to a function that wakes a list of waiting threads - as i got rid of that, i had nothing to lower the ipl again

still to be done is logic to send threads to different CPUs however, they all just go on BSP right now

tomorrow i will change that to do something suitably simplistic to start with

[neat steppe]

I've managed to implement fine grained (per processor and per object) locking so that might be useful

Not all the bells and whistles are in yet but the core mechanisms are there and "seem" to work lol

I tested with 24 threads and an array of 8 event objects, each thread has two indices i,j which start at its number modulo 8 and that plus 1 respectively; it sleeps on the event at index i and then signals the one at index j, increments i by 2 and increments j by 1 and repeats

When it wakes the event there's a priority boost of 2

This seemed to test all the codepaths pretty well including priority decay on quantum end after I added every two passes around the event table it spins for a few quanta

And it didn't deadlock or otherwise die

For hours ! So it seems decently solid, knock on wood

https://github.com/xrarch/mintia2/blob/main/OS/Executive/Ke/KeDispatch.jkl

@hexed solstice if you really want to be fancy you should also look at the ULE sources and see how they lay out their cpu topology structures and use them for affinitization where it tries to enqueue threads to processors in order of increasing "distance" from the last core it ran on; same L1, then same L2, then same L3, etc; this isn't super important for like 4-8 cores but is absolutely essential for systems with lots and lots of cores

I'm unsure how dfbsd accomplishes that if it never directly inserts a thread in a remote ready queue

[hexed solstice]

it's a very interesting problem which i would like to have a good solution to at some point

for NUMA particular care needs to be taken since it's all good and well localising as much to your NUMA domains as possible but all goes to waste if your threads migrate between them frequently

but equally it is better to run on the "wrong" NUMA domain than not at all

[hexed solstice]

something is broken and i'm not sure yet what

whatever it is has been revealed by or caused by the changes to the scheduler

and manifests with the posix server (or rather the stub of one) faulting on address 0xfffffffffffff000

seems to be consistently in

   0x4666b7 <_ZNSo6sentryC2ERSo+23>:    mov    -0x18(%rax),%rdi

only happens sometimes

%rax = 0 for whatever reason - i wonder why

[final hatch]

Well what is this _ZNSo6sentryC2ERSo

weird name to be showing

[hexed solstice]

std::ostream::sentry::sentry(std::ostream&)

[final hatch]

Weird thing to be crashing inside of

[final hatch]

*I think it'd have been better to show the demangled name but I digress

[hexed solstice]

oh this is very interesting

   0x00000000004666aa <+10>:    mov    (%rsi),%rax
[... blah blah blah ...]
=> 0x00000000004666b7 <+23>:    mov    -0x18(%rax),%rdi

rsi = 5049312 and (gdb) print (void*) 5049312
$4 = (void *) 0x4d0be0 std::cout

[final hatch]

maybe this [... blah blah blah ...] is resetting rax to 0

or maybe the value at rsi when executing 0x4666aa was 0

have you tried printing *((void*) 0x4D0BE0)

[hexed solstice]

it isn't, and the value at 5049312 is indeed (gdb) print (void*)5049312
$3 = (void *) 0x0

[final hatch]

can you use `` please, discord is messing up the formatting

[hexed solstice]

the most likely cause is probably a vmm disaster

to keep it well tested i subject everything to the severest paging dynamics

[hexed solstice]

#0  0x000000000040846e in std::ios_base::Init::Init() [clone .part.0] ()
#1  0x0000000000404cce in _GLOBAL__sub_I.00090__ZSt3cin ()
#2  0x00000000400159b0 in doInitialize (object=0x17fc00) at ../../../../mlibc/options/rtld/generic/linker.cpp:946
#3  0x0000000040018128 in Loader::initObjects (this=0x2c350) at ../../../../mlibc/options/rtld/generic/linker.cpp:1529
#4  0x0000000040009d11 in interpreterMain (entry_stack=0x2ff50) at ../../../../mlibc/options/rtld/generic/main.cpp:525
``` this is where it gets given a non-zero value - a watchpoint here fires at this point, if the thing is going to work

if it isn't going to work, that watchpoint doesn't fire

[hexed solstice]

occurrence of this problem is reduced (?) but not eliminated by turning off working set trimming

which suggests a scheduling interaction of some sort or another

that's odd

never mind, no it isn't

[final hatch]

maybe just a data race

[hexed solstice]

yes, i forgot to lock around prints from that path - unfortunately the problem is harder

[final hatch]

no_prepared == 16 is what I'm getting

[hexed solstice]

should be nio

i have filled the fault handler with printfs to no avail

[final hatch]

did you try progressively disabling VMM features such as pageout?

if you have it

[hexed solstice]

i tried disabling the working set trimmer which reduced the frequency but didn't abolish the problem altogether

but right now i am having all kinds of weird shit going on

at ../../../../kernel/dev/iop.m:163 (iop_continue):
kernel assertion failed:
        iop->stack_current != -1 && iop->stack_current < iop->stack_count

should not happen

it means something is rotten

that's in the virtio 9p port driver

i don't know how the iop stack pointer got to be -1

it shouldn't be

the stack traces reveals that the iop during its queueing was actually completed before the queueing function returned

but that is something i explicitly support

iop_continue() is written in such a bizarre way precisely because something like this can happen

the question is what i've done wrong to cause this

[neat steppe]

you forgot to say i love you too last time your mother called

[sullen fern]

what's an IOP?

[neat steppe]

IO packet

[sullen fern]

ah like windows irps?

[neat steppe]

yeah

[hexed solstice]

do you know what

it probably helps to save fpu state if my threads are migrating between cores a lot

i don't necessarily think that is the problem but i think it's a distinct possibility so i'm going to check

[final hatch]

well you shouldnt have to do that

if you disabled SSE and SSE2 then the compiler should only generate code with generic registers

[hexed solstice]

mlibc is using SSE registers extensively

[final hatch]

oh wait

user mode

yes definitely save FPU and SSE state in that case!

[hexed solstice]

now to try this repeatedly a few dozen times till i'm satisfied

i didn't consider it until i was poring over the disassembly of the relevant functions that should be setting up std::cout

only after spotting some incidences of playing with xmm0 and such i considered the possibility

[vale pelican]

that doesn't quite explain why the iop code broke though no?

[hexed solstice]

that one it doesn't

[neat steppe]

iop one was the less scary one anyway

[final hatch]

No but it might explain the other issue where some point is randomly zeroed

[neat steppe]

since its probably a relatively simple logic issue

[hexed solstice]

after adding fxsave/fxrstor i have not yet been able to reproduce it YET (i couch my words carefully, i won't tempt fate)

that's the cout = 0x0 one i can't yet reproduce not the iop one

that has popped up again once

[hexed solstice]

yes, that one i think it might actually be fine for stack pointer to go to -1 while going up the stack again after completion, i don't know for sure, i don't know how i managed to produce that ridiculously complex function in the first place

i think some muse possessed me

well this is a big relief in any case - i was not especially keen on the possibility that there could have been something wrong with the vmm, with the iop dispatcher, with the scheduler/synchronisation, or with some combination of the three

that would've threatened to be one of those bugs that makes you want to rm -rf the entire source tree

[neat steppe]

when id run my forkbomb stress test that would usually lead into single sitting 18 hour debugging sessions

sometimes followed by another 6-10 hours the next day until it could forkbomb for a few hours straight w/o crashing or leaking memory or anything

not all on one bug but on like 100 little bugs

often memory leaks and so on and not anything that would crash

[hexed solstice]

with this now, then, the dispatcher lock is now completely smashed (i smashed most of it last year with per-object locking, now its residual role as a global scheduling lock is abolished too)

[hexed solstice]

i can tolerate a memory leak, those at least you can see there's something you've not freed

for old keyronex i put together an interface to dump allocations and frees + where they were made from, and a script to look over them, find unfreed allocations, and addr2line on the allocator for each

then just launched programs from bash, exited them, and looked over them

there was some amount of noise (mostly because of the CoW'ing, that alters the state of things quite a bit, but mostly just the vm_anon_t structures - which i knew were no problem so i could just quash prints of those)

and although RCU sounds scary i actually expect to run into less bugs with RCU solutions i've started using than i was with previous approaches i've taken

[sullen fern]

yeah right now I think I leak about ~30 pages on program exit

[hexed solstice]

since within an RCU critical section i have a guarantee that pointers to RCU-protected structures won't see the object they point to go away

[sullen fern]

but that used to be ~1100

I think it may have to do with vmem

[hexed solstice]

extra boundary tags floating around?

[neat steppe]

right now i only use about 30 pages total

lol

[hexed solstice]

you should open a thread here about gaia btw

[sullen fern]

yeah I've been wanting to do so for some time now, I even brought back the old console

[hexed solstice]

i think it will be more appreciated than the thread at you-know-where

[sullen fern]

[hexed solstice]

just saw the latest post there, very nice

[final hatch]

if you dont mind, where?

[sullen fern]

open osdev

I wanted to troll a bit but that didnt succeed very much

[final hatch]

wazzat

[hexed solstice]

you don't want to know

[sullen fern]

I don't think I free segments properly

so yeah

[sullen fern]

i bought 16gb of ram might as well use 16gb of ram

[hexed solstice]

i am going to be using more memory than i might've been had i not adopted RCU

i'll have a look at FreeBSD's GUS some day, that's like RCU but it reclaims much faster

something else i should change promptly

i have things set up so %gs:0 is the current CPU - should become current thread instead

then accessing current thread can be an IPL 0 operation and only accessing current CPU need be an IPL >= DPC operation (inhibits scheduling)

[final hatch]

why can't getting the current thread already be an IPL 0 operation

i mean that thread isnt going to be executing when the currentthread of that CPU is different from the actually executing thread context

[hexed solstice]

because it's implemented like curcpu()->running-thread, and if you are scheduled away from at an inopportune time, and then you are scheduled back to on another CPU, the value given by curcpu() may no longer be true

[final hatch]

oh duh

youre right

this wouldnt be a problem if the current CPU was mapped at a fixed address but youd need to manage several page tables, one for each thread...

[hexed solstice]

one advantage of software refilled TLBs

you can do such as this trivially

[final hatch]

true

[hexed solstice]

now it's curthread that you access and then curcpu through that

nice and simple

this will save future grief i'm sure

[hexed solstice]

the time has come to implement rwlocks

and not just any rwlocks but fast ones

turnstile/pushlock style

[hexed solstice]

i have an initial design ready

now to implement it

it owes much to a description i've read of pushlocks

[hexed solstice]

i have done an initial implementation, untested yet. i will see whether i can test this in userland

[hexed solstice]

nah i've fucked it up a bit

now figuring out where i went wrong

[neat steppe]

we should never aspire to be only as good as reactos is

but i think they probably have the best open implementation of pushlocks

so you should check that out

[hexed solstice]

theirs is much better than mine, i just had a look earlier

much more sophisticated

ionescu did well

[neat steppe]

afaik its based on the windows implementation after a clean room reversal

so its probably as it was in windows in mid '00s

or whatever

i was going to base my first pass implementation of pushlocks on the reactos one

and add a "convert to shared" function as well

[hexed solstice]

theirs is a little mindblowing, i will need a bit of time to understand it

in any case mine falls apart and i am trying to chase down where

[hexed solstice]

at last

it's working now i think

[prime coyote]

what's changed

[hexed solstice]

i cleaned everything up

tried to avoid too much nesting of scope and fixed some bugs along the way

not releasing every wait block i should have was a big one

now seeing what threadsan says

only false positives with a 1000 reader/600 writer test (each sleeps for a random few milliseconds between acquisitions)

[zenith karma]

are you sure it's a false positive?

[hexed solstice]

i am positive it is the case

[zenith karma]

i think there exists some c++ tool that bruteforces every possible memory reordering btw

[hexed solstice]

if you find its name let me know

[zenith karma]

yeah i'm trying to find it

[hexed solstice]

test.c:221:25: warning: ‘atomic_thread_fence’ is not supported with ‘-fsanitize=thread’ [-Wtsan]
221 | __atomic_thread_fence(__ATOMIC_RELEASE);
oh that's why it complained

[zenith karma]

oh, yeah

fences don't work with tsan

something something it's really hard because of the way tsan works

(tsan requires that you synchronize against "something")

[sullen fern]

is tsan runtime?

[zenith karma]

yeah

[sullen fern]

I just found out clang has -Wthread-safety

[zenith karma]

no idea, my google-fu isn't good enough

[sullen fern]

How CBT is it to implement

[zenith karma]

massive, afaik

[hexed solstice]

does that one work for C now?

i remember seeing it before but it's C++ only

[zenith karma]

or actually, maybe not

[sullen fern]

Idk I use C++

[hexed solstice]

i was still considering adding locking annotations for the future

[zenith karma]

hmm. might be not too bad

i think the idea is something like

[sullen fern]

I would assume it's only C++ because it uses classes to represent locks

[zenith karma]

"store which version of memory writes are you guaranteed to be after"

[hexed solstice]

i am fairly confident this thing works now anyway but when there's a riscv board that isn't a pile of proprietary junk with a trash fork of edk2 that has bare minimum to boot linux from an sd card and no more, i will try this out there

[sullen fern]

But wouldn't it be too hard to adapt to C? Just change Mutex::lock to mutex_lock(&mutex)

[zenith karma]

and then if you do an unsynchronized read then whatever wrote it must have a lower global order than you

or else bug

this has massive overhead btw

[sullen fern]

ubsan has kind of a big overhead too from what I've tested

[zenith karma]

like asan is 8:1

and tsan is 1:8

[sullen fern]

Removing ubsan makes my OS much faster

[zenith karma]

memory overhead that is

[sullen fern]

no performance too

[zenith karma]

performance also gonna suck

because tsan is doing a lot more stuff

on every single load

[sullen fern]

yeah makes sense

[hexed solstice]

fun pushlock use described in win internals: an array of as many pushlocks as CPUs, acquire every single last one of them as writer to do writes, acquire only your local one as reader to do reads

[zenith karma]

i mean

[sullen fern]

Ugh I should learn more about synchronization now that I'm porting to smp

[zenith karma]

at this point just have one lock for writers

none for readers

and rcu

[sullen fern]

I only have spinlocks

mutex shouldn't be a bad place to start I guess

[zenith karma]

(well, i guess that includes an rcu critical section, but that's cheap)

[hexed solstice]

not awful if you write once every few minutes or so, not great if you do it more frequently

[hexed solstice]

yes, i have rcu now and frankly it will almost always be the better option

[zenith karma]

i have an idea for a slightly cursed trick to make rcu hopefully less expensive now tbh

[hexed solstice]

what i can't do since i don't have sleeping RCU is to have an RCU crit sec while dealing with pageable data or needing to sleep

but i am not anal-retentive about making everything pageable

[zenith karma]

always perform rcu load-and-reference in a special block of code well-known to the scheduler

so that when rescheduling you can force it to be atomic

[hexed solstice]

this is an interesting way to get rid of even the cost of disabling preemption

[zenith karma]

it has the cost of a call on every one of these atomic loads

which has nonzero cost

because you can't inline them unless you REALLY hate yourself

although

[hexed solstice]

this is what i am less keen on, in most cases the dereference and refcount ++ can be inlined

[zenith karma]

you can

you just need an instruction decoder in your scheduler

there's exactly one scenario that is unsafe

load; reschedule; increment

and an increment always has the bit pattern f0 4[89cd] 01 [0-3][0-f]

on x86 anyway

so if this is the instruction at rip, you can trivially skip it

on arm, it's either an external thunk, so you already pay the price of a call

or ldadda

which seems to be kinda new

actually armv8.1 isn't that new

on arm, it's always f8[ab][0-f]0[0-3][0-f][0-f]

(or a thunk)

seems like it's a thunk on m68k too

[hexed solstice]

i have a raspberry pi 4 arriving today

i was going to try to port to a RISC-V board but somehow most of them are even worse than a pi in terms of being weird proprietary junk with at best the bare minimum done to permit booting linux

the aarch64 port will need to be resuscitated, i haven't built it in 9 months

everything that could change around it has changed

[twin haven]

exciting, keen to see how this goes

I'm thinking of aarch64 for my next port

is the pi 4 hardware documentation any good?

[hexed solstice]

the per platform virtual memroy support code shouldn't be too hard to get into shape at least - for amd64 it is about 400 lines, a lot of which is struct definitions because the actual logic for dealing with multi-level page tables is now portable generic code

[hexed solstice]

well, so far i found no formal documentation on the ethernet controller, but it's at least got several drivers written for it

unfortunately it's called genet which is being used as the title of several papers on AI (you know, neural networks, so google can get a bit confused)

https://datasheets.raspberrypi.com/bcm2711/bcm2711-peripherals.pdf there is this little sheet which says a few things

[twin haven]

ah excellent, ai is everywhere 🤦‍♂️

[hexed solstice]

oh and i remember @vale pelican saying once that pcie dma is cache incoherent on the pi 4

[vale pelican]

yeah it is

i don't think it's the only arm soc where that's the case, linux has a dma-coherent dt property to tell ypu whether it is

[vale pelican]

paging in general is quite similar to x86_64

one annoyance is that you'll need to emulate dirty/accessed bits yourself (the latter via a special flag that causes a special fault if unset)

newer aarch64 cores can do dirty/accessed in hw, but v8.0 that the pi4 is doesn't (and it doesn't implement it as an optional feature either)

[neat steppe]

nothing generally can do that except x86

[neat steppe]

until recently ig

[twin haven]

riscv can do it, optionally - but there's no way to know if it'll page fault (with those bits being clear) or set those bits until it happens.

[neat steppe]

it was generally considered a bad idea because of how much it complicates the paging logic and cache interactions

but with more die space (or rather smaller features) comes more freedom to do ideas that used to be bad

[twin haven]

tbh I have no idea about that stuff, I just know software man

but I appreciate it being a feature

[hexed solstice]

i don't use the accessed or dirty bits at all at the moment

accessed bit i do want to make use of in the future but i can have a shorter fault path for that case

[hollow hull]

my suggestion is this if you wanna port to rv64 or aa64

use the Limine protocol lol

don't port to boards without a proper UEFI directly

you're just wasting your time

and feeding into the embedded dev brainrot cycle

[neat steppe]

a big part of this is discussing this exact thing https://yarchive.net/comp/software_tlb.html

[hexed solstice]

this is why i ordered a pi 4 and not one of those much vaunted riscv boards, those starfive/sifive whatever they're called

[valid sequoia]

good idea, and pi5 is weird and expensive and honestly meh

[hexed solstice]

they are hard to distinguish from just another weird proprietary embedded thing

[hollow hull]

alternatively if you had money to waste you could've even bought one of those qualcomm laptops :^)

[valid sequoia]

you can buy human organs for that much money

[hexed solstice]

i am not exactly delighted with the pi either in this regard but at least it's very popular and does have a proper edk2 port so limine should just work

[hollow hull]

i wonder using Limine on aarch64 with how much portable code you can get away with, that doesn't need special hardware support

like, take most of our OSes, they work pretty much unmodified on most x86 PCs

i wonder how much you can apply that to semi-standard UEFI-bearing aarch64 machines

[valid sequoia]

you would only need pcie and usb mainly

[hollow hull]

well but that goes for x86 too

[hexed solstice]

it's an open platform and i don't think people appreciate this enough

[hollow hull]

i meant something like special hardware that you need to implement for basic functionality for each board/machine

[valid sequoia]

on modern aarch64 no

excluding apple and pi3

[hollow hull]

what about the gic vs whatever apple uses?

[valid sequoia]

apple is the only exception apart from early broadcom

[hollow hull]

hm, well it's quite a big exception

[hexed solstice]

nowadays netbsd at least has a single port (Evbarm) and a single binary for that boots most aarch64 boards

[hollow hull]

and if it's just the gic vs whatever apple uses, it doesn't sound too bad

assuming the rest would work fine on apple

[valid sequoia]

mostly

[hollow hull]

mostly

[valid sequoia]

they are a bit special compared to the rest of the arm world but not spec breaking

like using fiqs

[hexed solstice]

pi is here and edk2 set up, surprisingly painless

i am not hurrying to port keyronex yet though

there is the beginnings of a port i started about a year ago but needs much work to adapt to changes

[hexed solstice]

uacpi took its sweet time to talk

but that's the old port now apparently working again, well, to the little extent that it does anything

and plenty of times qemu just does

BdsDxe: starting Boot0001 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x2,0x0)

Synchronous Exception at 0x000000004761C978

Synchronous Exception at 0x000000004761C978

[sinful jetty]

Wait thats on aarch64?

[hexed solstice]

yes, i don't know if you target it yet

[sinful jetty]

Damn

[hexed solstice]

but it did that

[sinful jetty]

Yeah I do, but you're the first person to ever run it on aarch64

[hexed solstice]

on the real hardware it took about 10 seconds to complete namespace initialisation, i don't know why yet

[sinful jetty]

Cache issues maybe?

[hexed solstice]

oh, congrats on your milestone then

[sinful jetty]

Yeah thanks im impressed

[final hatch]

What AArch64 device are you running it on?

[sinful jetty]

Do you support shutting down? Because thats reduced hardware shutdown and I've never tested that

[hexed solstice]

it's possible, i am still providing it HHDM pointers which is simply not going to fly on arm

[sinful jetty]

Ye

[hexed solstice]

there is no pci config read/write at all yet and uacpi doesn't appear to try

[hexed solstice]

not yet but i can try it once i clear up some big problems in the way

[sinful jetty]

Would be cool to get a dump of those tables

[hexed solstice]

raspberry pi 4

[sinful jetty]

Yeah that would be great, you're the only person with uacpi to support aarch64 in acpi mode

[hexed solstice]

i was going to target riscv64 first but riscv64 boards are all stupid trash

some of them do have hacked up monstrosity edk2 forks running on them but they never get updated and are crippled

[final hatch]

why? caching issues?

[hexed solstice]

god help me

[sinful jetty]

Lol

[hexed solstice]

yes, arm is not as friendly as x86

[final hatch]

makes sense

[final hatch]

i mean arm is another risc

acorn RISC machine

[neat steppe]

https://tenor.com/view/trump-wow-telling-me-now-first-time-surprised-gif-1695387857546901797

[hexed solstice]

did you know that stupid cow theresa may sold it off?

she said, and i quote, that selling off ARM holdings to softbank (whoever they are) showed that "Britain is open for business"

[neat steppe]

more like a foreclosure sale

[hexed solstice]

shows it's open for business in the sense of seeing a prostitute get into someone's car

but at least they don't have to actually lose something in the exchange

[zenith karma]

softbank is some vc, afaik

[hexed solstice]

embarassing

[zenith karma]

they are the people who gave literal billions to wework (shared coworking space that went out of bussiness because their business model is well known to be bad)

[hexed solstice]

forget energy and rail - labour should nationalise ARM

[zenith karma]

that would be rather based ngl

but arm is like

publicly traded

[hexed solstice]

sophie wilson to be installed as head of industrial strategy

BdsDxe: starting Boot0001 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x2,0x0)

Synchronous Exception at 0x000000004761C978

Synchronous Exception at 0x000000004761C978
this bloody thing again

can't even get into limine most times i try to boot with bloody qemu

i think i had a bunk ovmf image

[vale pelican]

for qemu the ovmf image that limine downloads should work fine

although iirc it could also not work well if you have an older version of qemu

[hexed solstice]

i have fetched that one and it works fine

the one i had before was trash

[hexed solstice]

one question btw (i think you invented limine for aarch64) - why is PSTATE.SP = 0?

[vale pelican]

iirc its because with spsel=0 in the kernel you get a separate irq stack for free in kernel mode

since when entering an exception handler it automatically switches to spsel=1 (and then eret switches back)

(spsel being the system register that exposes the state of pstate.sp)

switching between one or the other is not that problematic though

[hexed solstice]

that's fair enough i suppose

i have to admit to some surprise that keyronex/aarch64 didn't require that much editing to get back to where it was before

[vale pelican]

nice

[hexed solstice]

it is still very primitive, no IPL, no timers, no task switching (there is code, which looks like it might work? i don't remember writing it), no user mode, etc

[violet tinsel]

wow, i‘m impressed, well done

[hexed solstice]

and since i want to clean this area up anyway i am thinking of replacing the separate entry points for amd64 and aarch64 with a common one

[vale pelican]

should be simple enough if you use the limine proto for both

[hexed solstice]

it might even become a common one for m68k too as i can move out the areas where it directly touches limine responses

early startup needs a careful sequence of initialisations, a little here, then the rest later

setting up the first thread and bootstrap cpu struct and such

[hexed solstice]

thanks, but it was mostly just dredging up what was there from about a year ago

what i did like is that the generic page table wrangling logic appears to have just worked

[violet tinsel]

Still one of the greatest projects here imo : )

[hexed solstice]

it's already equipped for separate user/supervisor page tables because of the 68k port

[vale pelican]

btw re pcie on the pi4, i think if you use acpi it exposes xhci directly as a device in aml, and does not expose the pcie controller

because the pcie controller can't be exposed via MCFG (config space io does not follow ecam)

and i don't see the linux driver for the controller matching against any acpi pnp ids

[hexed solstice]

this explains why uacpi didn't call one the instant-panic stub pci conf access functions

[hexed solstice]

so i have a doubly uncharted path to walk

uacpi on real aarch64 for the first time and the free OSes don't make a lot of effort with acpi on pi 4s at all

[hexed solstice]

thanks, i am glad it's appreciated

[final hatch]

hey, did the master branch of Keyronex reach userspace?

just wondering

[vale pelican]

well at least it saves you from having to write a driver for the pcie controller and code to do things like bar allocation

unless you decide to also add device tree support later on

[hexed solstice]

it does but only on m68k and amd64, and it doesn't do much there

[hexed solstice]

of course i'd love to add device tree support...

but not ere i run on powerpcs or even sparcs

and if i can't call fcode methods on the devices in the tree i don't want to see it

[vale pelican]

one thing i wonder about is how the aml deals with dma address limits for devices

i guess _CRS or such?

this is kinda important on the pi4 because dma can't go above 3GB

although by default uefi just limits the amount of memory to 3GB unless you change the option in setup

okay after some digging in the linux code, it checks the IORT table, and just does min(all the dma limits) to find the max phys addr for dma

afaics anyway

[hollow hull]

nice job

[sinful jetty]

Would be interesting to see

[hexed solstice]

wtf keyronex/amd64 is fast now

and this is in spite of it doing many TLB shootdowns

maybe qemu sped up

[sullen fern]

Was it slow beforehand?

[hexed solstice]

on SMP terribly

but it's changed, i think since i've upgraded to newer qemu

[sullen fern]

Why?

Is it faster on non smp

[hexed solstice]

i deliberately impose limits to force extensive swapping of pages

it keeps testing the vm code

[sullen fern]

ah

Apparently a crappy scheduler can make smp slower than single core too

According to Marshall k. Mckusick

[twin haven]

very cool, congrats!

[twin haven]

excuse the self-advertisement, but I do have a (very) primitive limine protocol loader for m68k you're welcome to use if you want. I wrote it for myself so I could keep using the lbp across all archs.

[hexed solstice]

cheers

[hexed solstice]

i have been considering this

and i will likely be moving over to either that or to rooster

[hexed solstice]

i am moving just as slowly as i expected

basic gicv2 support is underway, as is interrupt priority management (since adjusting current priority is slightly less conveniently accessible with the GICv2 than it for the APIC on x86, IPLs <= DPC are emulated)

[glad sentinel]

maybe this can be another kick up my backside lol

though lately I've had compilers on the brain

[hexed solstice]

it's not much, but now that i'm starting to fill in the per-arch dependencies (today threading related stuff) we move into the next stage of startup - entering the executive. the two daemons that deal in page replacement are ip, and happily go to sleep

888 active pages, lucky number

looks like aarch64 is one of the blue platforms (so is amd64, m68k is black)

[vale pelican]

beware that the icache is not coherent with the dcache on it's own

so you might need extra cache maintenance code when faulting in executable pages

[fierce blaze]

so blue is 64-bit while black is 32-bit?

[hexed solstice]

will serve me well if i ever port to other RISCs also

[hexed solstice]

it's more of an endian problem i think

or maybe a virtio-gpu thing (that's what keyronex for qemu m68k "virt" uses)

i have not bothered to fix it because it adds character to the system

[sinful jetty]

are you cooking something up 👀

[hexed solstice]

something, but not keyronex related

it involves a star that's particularly prominent in our sky

[sinful jetty]

A new secret project?

[final hatch]

Polaris?

[hexed solstice]

you are off by a letter

pardon

[hexed solstice]

you are off by a letter

[sinful jetty]

lol

i was sitting there for a second wondering which letter

[hexed solstice]

looks like i've been caught

[final hatch]

so what's with Polari

[hexed solstice]

i'm being facetious, it's solaris

[final hatch]

Ah lol

so.... what is with Solaris

[hexed solstice]

you'll find out in due time

[sinful jetty]

why acpica tho?

i have been betrayed

[hexed solstice]

solaris relies on it and i am not rewriting it to use uacpi

you are welcome to try

[sinful jetty]

ah

once i find out what it is

[final hatch]

cool, good luck

[hexed solstice]

i will just admit it, since there is nothing to have posted yet, a late solaris fan called joerg schilling used to relentlessly claim that solaris was a microkernel

and i have a funny idea involving misrepresenting what "microkernel" means and implementing a zone brand (a technique that provides binary compatibility with other kernels) for netware nlms

[sinful jetty]

Damn

[hexed solstice]

it should make for a funny reaction from the tech press and forums

i do have new developments on the keyronex front though

i am expecting to add support for large pages

[sinful jetty]

Nice

[hexed solstice]

what work i have done so far is only available to the kernel for wired mappings as i don't know how to manage them effectively for pageable userland mappings yet

they raise too many questions

if you have freebsd/linux style transparent large pages then you are effectively promoting areas of memory to permanence, because it's a lot harder for the accessed bit to stay 0 for a 2mib page than a 4kib page

you might fairly decide on what to promote to 2mib by monitoring the access patterns to the area you're considering promoting and if most of the 4k pages are frequently accessed then go ahead and make it a large page

but access patterns change over time

[hexed solstice]

that reminds me, kindly add the GTDT definition to acpi.h when you have the time

[sinful jetty]

Will do

[sinful jetty]

THP style fs of course

But tbh a madvise call is good enough I think

Let user decide when they want large pages

[hexed solstice]

a user on the osdev irc was speaking positively of linux transparent hugepages as "essential for some workloads" whch surprised me because a) if your workload needs it, you should frankly be asking for it, not hoping the kernel does it for you, and b) there are THP-related regressions reported everywhere, a huge number of people turn them straight off

and i think it is actually quite a hard balance to strike between virtual memory as an invisible thing in the background that tries to do what's right, as opposed to one that wants lots more information than most programs are willing to give

he basically glossed over the problem of a large page tying up 2mib when just a single byte is accessed of it from time to time, but that is a real problem, and also makes it harder to get other large pages put together

[sinful jetty]

I agree yeah

[hexed solstice]

i am thinking of adding proper per-cpu data support as well

yes, i think it will work for all arches i currently or intend to support (amd64, m68k, aarch64, and future riscv64)

[final hatch]

Isn't that just the processor block/CPU struct

[hexed solstice]

if you point GSBase at it you can then do mov %gs:offsetof(some member), %whatever and load something from it without having to inhibit preemption

but you can go further yet

[final hatch]

but you have to inhibit preemption when accessing it anyway don't you?

[hexed solstice]

you can arrange to have CPU local variables go into a special section, then when allocating the processor block, allocate as much extra bytes as that section holds, and then you can load per-cpu variables declared anywhere by finding the offset of the per-cpu variable from the start of the per-cpu section and using that + the sizeof the static part of the processor block as your offset

[final hatch]

hmm

[hexed solstice]

for most i think realistically yuo would have to

[final hatch]

I wonder if I should just have a KiCurrentPrcb which is a symbol defined such that it's accessed using gs: instead of ds:

[hexed solstice]

one thing useful you can do without inhibiting preemption, incrementing performance counters

[sinful jetty]

oh interesting, it's arm specific

going to add it in now

[hexed solstice]

doing a bit more aarch64 stuff today, unsurprisingly

it's a nice time to abstract IRQ source management

[final hatch]

@hexed solstice i was thinking of implementing some kind of "block device proxy". Basically it would turn unaligned/whatever writes (think, read(64, 2000) for example where 64 is the beginning byte offset and 2000 is the length) into something like read(0, 512) (would fill out a 448 byte region), read(512, 1536) and read(2048, 512) (would fill out a 512 byte region) in the case of 512-byte-block devices

i wonder if this is necessary though.

[sinful jetty]

why not just let it fill the page cache for these blocks and copy whatever is needed into userspace from there?

[final hatch]

Actually I wonder if it's feasible to completely restrict non-page-size-aligned writes at all directly to the driver

and then regular standard I/O would be performed via the page cache

or filtered as mentioned before if the device forbids going through the page cache

[sinful jetty]

yeah a block device should only support aligned writes

[final hatch]

but I mean page size aligned

so like itd read 4096 byte bursts (8 512-byte blocks) rather than specifically the device's block size

[final hatch]

what do you think of that?

[sinful jetty]

Maybe that's okay

[hexed solstice]

nice progress, needs page fault handling to go any further

[hexed solstice]

i think that's proper for most disk access

some questions to consider though

the "block device" is a unix concept (absent from some modern unixes) - it provides buffered I/O

coherent with buffer cache data used by the filesystem - that's why you can fsck a live filesystem (well, a read-only one)

do you need that?

your FS drivers will obviously read whole blocks at a time (and file data usually in pages' worth, even if discontiguous on disk)

freebsd got rid of block devices altogether and only has character devices now (for character devices, you have to make block-aligned, block-multiple read()/write() requests)

[final hatch]

so hard drives and shit are implemented as char devices?

[hexed solstice]

yes, they got rid of the traditional buffer cache so block devices became obsolete

i will not be supporting block devices either as i can't see a use for them

character devices are adequate, i don't expect to have a traditional buffer cache so supporting block device-style random-access to any unaligned bytes on a disk is needless

[final hatch]

I thought block devices were just I/O devices which operate on blocks

Is there anything more to it in the unix concept?

[hexed solstice]

they're devices that are backed by a buffer cache and use that to provide unaligned random-access to any bytes on the disk

while the character device representing a disk is strictly limited to block-aligned I/O

[final hatch]

and i thought character devices were like

terminals and serial ports

and keyboards

[hexed solstice]

they're also hard drives, CD-ROMs, etc

[final hatch]

so what examples of block devices are there?

[hexed solstice]

hard drives, cd-roms etc are block devices too

except on OSes which abolished the concept of block devices

[sinful jetty]

Thats news to me as well

Are you sure its not the other way around Fadanoid?

[hexed solstice]

completely sure

[sinful jetty]

Character devices only supporting block size access is really strange

[final hatch]

I suppose you can define the "block" here as 1 byte in the case of serial ports, keyboards, and terminals

¯_(ツ)_/¯

[hexed solstice]

incredibly wikipedia actually describes this correctly:

Block special files or block devices provide buffered access to hardware devices, and provide some abstraction from their specifics.[7] Unlike character devices, block devices will always allow the programmer to read or write a block of any size (including single characters/bytes) and any alignment. The downside is that because block devices are buffered, the programmer does not know how long it will take before written data is passed from the kernel's buffers to the actual device, or indeed in what order two separate writes will arrive at the physical device. Additionally, if the same hardware exposes both character and block devices, there is a risk of data corruption due to clients using the character device being unaware of changes made in the buffers of the block device.

Most systems create both block and character devices to represent hardware like hard disks. FreeBSD and Linux notably do not; the former has removed support for block devices,[8] while the latter creates only block devices. To get the effect of a character device from a block device on Linux, one must open the device with the Linux-specific O_DIRECT flag.

[sinful jetty]

Oh okay now it makes sense

[hexed solstice]

the citation 7 is to the posix spec which says little:

3.79 Block Special File
A file that refers to a device. A block special file is normally distinguished from a character special file by providing access to the device in a manner such that the hardware characteristics of the device are not visible.

[sinful jetty]

On Linux its always a block device

https://www.linuxjournal.com/article/2890

This looks like good reading for later

I like the O_DIRECT approach better than having two separate nodes

It feels cleaner

[hexed solstice]

i keep getting an assertion tripped in the IOP continuation dispatch routine

i cannot figure it out yet

aarch64 port, at first i thought it was to do with iops not being locked, and maybe an interrupt arrived untimeously on another core and there had been some reordering of memory

but it's same core and seems to persist despite a compiler barrier in the needful place

now i start to wonder if the assertion should be commented out

because things seem to work if it is

nope, it's needed#

i have bigger problems in any case

random little crashes popping up everywhere

some sort of state corruption i can only speculate

[violet tinsel]

if you had written your kernel in rust this wouldn’t have happened 🤓

sorry i had to

Whats the assertion that triggers about? some corrupted data structure?

[hexed solstice]

bad iop stack pointer

iop stack pointers transiently go to -1 (they start there because the first continuation that starts it going bumps it up, and they also end there, because the value gets bumped down when moving back up the stack, and that's the termination condition)

i am seeing -2

i was expecting some potential problems anyway because of aarch64 not being TSO and there being no appropriate barriers in there, and if someone then does something that will cause the iop to be continued by someone else (i.e. an interrupt signalling completion of a request), and that happens on another core, but the iop stack pointer hasn't flushed first, you're in trouble

but this is on a uniprocessor

and also happening in qemu running on amd64 host

iops are not locked, i should mention, because that would be an intolerable overhead, and because they are effectively in the single custody of one person at all times

anyway that's for dealing with after i figure out why things are rotten

it will turn out to be something trivial probably

in a position where i expect to see x0 containing a particular value i see none and i suspect messed up task switch or exception code

but i am winding down for bed now and can't chase it any further sadly

hold on i see it

that's after the bloody general registers are restored

hallelujah, now it crashes where it's supposed to crash again

[hexed solstice]

something i want to implement in keyronex in the long term

per-table locks on page tables, and on the multi-level tables that files/anonymous objects store their pages in

i want variable-depth radix trees for objects

the first level will be two or so pointers in the object itself, protected by its own special lock, these would be for optimising the case where there's e.g. a few hundred to 8 kib of data in a file and it would be unseemly to allocate even a single 4k table page to have no more than 2 entries set (let alone 4 levels of tables as i currently do)

[twin haven]

ooh this sounds interesting

[hexed solstice]

back to the aarch64 work

i need to deal with traps from el0 and make a syscall interface

[final hatch]

what device you running this on?

[hexed solstice]

qemu

it does do basics on a raspberry pi too

[final hatch]

ok but what device are you emulating

[hexed solstice]

but i'll need to write a driver for the janet nic to get any further on the pi

keyronex on real hardware has only ever network booted and i would have to do some work to support a tmpfs boot

[hexed solstice]

it's the qemu "virt" machine

[final hatch]

ah

[hexed solstice]

that one is nice to test with because i could easily give it virtio devices on PCI

after implementing ecam mapping and interrupt routing the existing virtio-9p driver basically just worked

[hexed solstice]

oh, how handy, it looks like i already defined a syscall interface

[hexed solstice]

userland execution in keyronex/aarch64

[violet tinsel]

epic

[hexed solstice]

no SMP yet

but i think that should be alright

[final hatch]

damn nice

is this using the posix server?

[hexed solstice]

well

the "posix server" just spawns some threads and mmaps some stuff and runs around it

[final hatch]

i see

[hexed solstice]

oh, how silly of me, i forgot to turn on the el1 timer

now it gets to the third thread too

i'll have to try this under KVM on my pi4

there will probably be problems

this porting stuff is such a fun jaunt

especially when mlibc already targets the arch

riscv64 will be the next target

then i'll be the (tied with northport) most portable OS of the server

the aarch64 port weighs 1500 lines of code, i think riscv port might be smaller

on balance i think i like aarch64, i don't like the proprietary bs platforms that all aarch64 real hardware is though

[sinful jetty]

did u manage to wrap your head around all the insane cache settings, barriers, and the the tlb invalidation instruction?

[hexed solstice]

roughly, there is still some stuff to do for SMP

[sinful jetty]

especially the inter processor caching settings are just brutal

i know u just usually hardcode them to one specific thing but still

[hexed solstice]

linux for the longest time, i had a look, seemed to map every device ngnre

[sinful jetty]

yeah that one i did wrap my head around

[hexed solstice]

the shareability stuff is going too far frankly

[sinful jetty]

yeah that

inner/outer etc

just figure it out in hardware jesus man lol

[hexed solstice]

it's for obscure setups i think

you can fairly assume your kernel belongs to a single inner shareable domain

[sinful jetty]

i think last time i checked linux just hardcoded this

ngnre is also the usual thing

some devices need the even stricter one

but its either one or the other, no inbetween

aarch64 feels like vulkan of cpus honestly

and x86 is like opengl

even the way u configure the virtual memory bitness is more involved

but i know nothing about m68k maybe its also as involved

[hexed solstice]

it's not too bad

just very extra

movem.l  %d0-%d7/%a0-%a6, %sp@-

[valid sequoia]

even nintendo couldnt get it right

[hexed solstice]

the 68030 MMU is insane though

[sinful jetty]

wdym

[sinful jetty]

lol

[valid sequoia]

the switch os had a barrier bug for the longest time

[sinful jetty]

should've used an x86 apu

[hexed solstice]

bringing up SMP now

[hexed solstice]

i didn't do it that quickly, i started a little earlier

and this is still not on qemu -enable-kvm on the pi

let alone on bare metal

[vale pelican]

iirc linux just sets everything to be inner shareable

also i think linux recently gained a way to map something as device nGnRnE instead of only supporting nGnRE

i think it was due to asahi linux

because on apple silicon you needed to use nGnRnE for something or it'd just raise an asynchronous exception (or do nothing? idr exactly)

[hexed solstice]

looks like i can't get anything to happen in qemu with kvm anyway

edk2 gets nowhere

[sinful jetty]

Yup

I've looked at the code when writing arm support for hyper

And Linux indeed hardcodes that and the thing you mentioned for apple

[hexed solstice]

i give up

the virtualisation stack on esoteric arches like aarch64 is obviously not the robust thing we are used to on mainstream arches, like amd64, and m68k

[sinful jetty]

What happened?