Keyronex

[hexed solstice]

cheers, it's been good to have the motivation again

i watch your wine experiments with some interest

[hexed solstice]

tcp could grow to be one of the largest parts of the kernel

already is is 3100 lines without the persistence timer, without keepalive, without congestion control, without IPv6 support, without SACK

[hexed solstice]

this morning's work: congestion control

[earnest salmon]

are you implementing multiple algorithms

for congestion control

and making them configurable

[hexed solstice]

for now just new reno but i'm factoring most of the logic into functions on_dup_ack, on_new_ack, on_rexmit_timer etc so that this is possible in the future

[earnest salmon]

sweet

gl

[hexed solstice]

a first attempt is done, not fully RFC compliant

[earnest salmon]

who needed rfc compliance anyway

[hexed solstice]

i also figured out and fixed why you couldn't have a new remote X client after ctrl-c the last client

i gave POLLHUP alone when there are no writers to a pipe but should include POLLREAD too

xorg cancels its input thread that way, by closing the write end of a pipe

(when the last remote client exits, xorg wants to reset the server, tearing down the old input thread and making a new one)

[hexed solstice]

it's time to start work on multiqueue NIC support and RSS

[high solar]

very cool. Are you doing cpu affine queues or how are implementing it?

[hexed solstice]

cpu-affine if enough queue pairs available, in which case i can even get rid of locking in the nic driver altogether both for receive and transmit; otherwise as many as i can

there are cases if you have fewer queue pairs where you may want to fan out in software instead but that's to be explored later

[hexed solstice]

i've shimmed everything so i can test the networking stack on linux (with TAP multiqueue + an eBPF kernlet to direct packets to queues according to hash). since life is much easier with access to sanitisers etc

[sinful jetty]

In this case Linux spreads them every N CPUs

Basically num_cpus / num_queues

[sinful jetty]

Can you explain what you're trying to achieve?

[hexed solstice]

shim keyronex interfaces so the networking stack and supporting infrastructure (STREAMS etc) can run as a program on linux, so that i can much more easily debug and test it

[sinful jetty]

What kind of shim code is needed?

[hexed solstice]

mostly the kernel synchronisation objects and other things required by STREAMS, also minimal vnodes, file objects because they have important implications on lifetime, i want to test certain subtleties like the interaction between the NIC driver and the IP stack so i went to the extreme of a rough imitation of the IPL concept, DPCs, and the timers

i could have gone to an even further extreme - have one thread per cpu and schedule within it - that would have been interesting but complicates things

so i map keyronex threads to pthreads instead and model IPL as a hierarchy of mutexes

[sinful jetty]

Id be interested to see how you implement all of that in userspace

[hexed solstice]

the current setup is that a thread per keyronex CPU runs the timer and DPC stuff, another thread per CPU is for the 'NIC driver', IP and TCP spawn their own threads so they've got one thread per CPU for doing work on

[sullen fern]

I have been doing this from the start and it's great

I've been shilling it quite a bit but its genuinely awesome

[hexed solstice]

i'm spending a bit of time planning enhancements to the networking stack

this is a good time to plan for ipv6, radix tree based routing table, lockless fastpath, that sort of thing

[hexed solstice]

doing the new routing table now. a patricia tree

[high solar]

how do you do longest matching prefix queries on that? just multiple queries with an increasing number of trailing bits set to zero?

[hexed solstice]

starting with root node, test if current node prefix matches the beginning bits of the query, if it matches, let best = that node, then check if next bit of query matches the left or right child and descend accordingly, break when there is a dead end or mismatch, return best

[high solar]

Ah so it's a binary tree and the inner nodes store values as well

That makes sense

[stone orbit]

a binary tree keyed by the ctz(key)?

[hexed solstice]

it's like this

[stone orbit]

kind of like a trie?

or maybe a prefix tree

oh wait aren't those the same

[hexed solstice]

the distinguishing feature is the compressed structure

otherwise it wouldn't be very fun for a routing table

especially with 128 bits in an ipv6 address

this is all overkill for keyronex but i couldn't resist doing it

[stone orbit]

that is pretty cool

honestly writing your own kernel is overkill too

you could just read a book :^)

i mean, depends on why you are writing your own kernel

but i think most people do it either to learn about kernels or for fun

[hexed solstice]

it's a satisfying data structure and that's got to be worth something

freebsd goes much further and couples a patricia tree routing table with some kind of optimal data structure called dxr, which i do not know anything about, but it's under epoch protection and they use that (called the FIB) for actual routing decisions

[final hatch]

@hexed solstice hey, where did the name "keyronex" come from?

[hexed solstice]

chatgpt

[final hatch]

lol actually?

[hexed solstice]

back when it first came out and was cool and capable (it has never been as good as that, it went downhill very early in 2023)

i gave it, i think, a list of names i liked but which were used by other OSes, they included probably some with ix or ex or ux in its name, keykos was in there too

[hexed solstice]

No.    Time    Source    Destination    Protocol    Length    Info
1    0.000000000    fd00::1    ff02::1:ff00:2    ICMPv6    86    Neighbor Solicitation for fd00::2 from 1e:9c:85:e2:f2:e8
2    0.001723594    fd00::2    fd00::1    ICMPv6    86    Neighbor Advertisement fd00::2 (sol, ovr) is at 52:54:00:12:34:56

ipv6 support is underway

also planning for RCU-readable neighbour cache and trying to localise as many of the steps in input packet processing to a single context as possible

in principle you can do a STREAMS topology of {socket a, socket b} -> {tcp, udp} -> ip de/multiplexor -> {nic a queue 1, nic a queuepair 2, .. nic b queuepair 1} but the level of interknowledge between the l4 and l3 protocols is so great that it's a poor abstraction and you're better served having a dumb de/multiplexor and having a combined tcp/ip, udp/ip module

[sinful jetty]

damn nice

ipv6 is not easy

[final hatch]

isn't it pretty much just a structure difference

and maybe IP address negotiation difference

[hexed solstice]

there's a bit more to it

especially if trying to be even approximately RFC-compliant

[earnest salmon]

ARP is also different

u basically need to icmpv6 to even get it to work iirc

[hexed solstice]

they introduce ndp which is transported as icmpv6

it's subtler than ARP but i think i can implement a generic neighbour cache in terms of the state machine expected by NDP/NUD and then implement both ARP and NDP as consumers thereof, ARP treated as something a little like a 'subset' of sorts of the NDP/NUD semantics

[final hatch]

i see

[hexed solstice]

this evening's plan:

• ipv6 support for routing tables
• icmpv6 echo reply (should have first ipv6 ping this evening)
• maybe start working on DAD?

[earnest salmon]

what is DAD?

[obtuse falcon]

parent

[hexed solstice]

duplicate address detection

[sinful jetty]

father

[hexed solstice]

whenever an address is assigned you have to flag it as tentative and send a neighbour solictation (this is like an ARP probe) and set a timer

and if you get a neighbour advertisement (equivalent to an ARP advertisement) in that period, you have to disable the address

No.    Time    Source    Destination    Protocol    Length    Info
5    15.424842836    fd00::1    fd00::2    ICMPv6    118    Echo (ping) request id=0x000c, seq=1, hop limit=64 (reply in 6)
6    15.425583920    fd00::2    fd00::1    ICMPv6    118    Echo (ping) reply id=0x000c, seq=1, hop limit=64 (request in 5)

oddity in optimised build:

[root@keyronex /]$ ifconfig net0
net0:   flags=73<UP, RUNNING>
        inet 10.0.2.20 netmask ffffff00
        inet6 fd00::2/64
SIGXFSZ (core dumped)

i've never even heard of SIGXFSZ

nor for that matter do i implement signals in this version of keyronex

i must be returning nonsense to wait() somehow

(i was)

[hexed solstice]

this evening's plan: maybe implement more of the NUD state machine? there's supposed to be timers and stuff

[worn elbow]

this evening's plan:

• maybe start working on DAD?

next evening's plan: maybe implement more of the NUD state machine? there's supposed to be timers and stuff

👌

[hexed solstice]

NUD turns out to be a prerequisite

[hexed solstice]

DAD is on the way

[earnest salmon]

should I get him a beer

although it is 350 on a Friday afternoon

ugh the joke doesnt work its friday

[tribal nova]

I'm not sure how to follow projects in GitHub. I starred it, but if there's a better way I'm all ears.

[sinful jetty]

There's a watch button

[hexed solstice]

i wonder how much dhcpcd needs to run?

it seems like it would make for a fun port

it wants BPF

[sinful jetty]

Lol

Bpf support in keyronex??

[zenith karma]

i found two (exploitable, i think) bugs in keyronex:

1. You can reach strwrite with arbitrary buf+len (e.g. through fifo_write); str_allocb allocates the backing using kmem_alloc; if size is extremely high (e.g. 2^64-1), roundup2(size, PGSIZE)/PGSIZE will return 0 and you will attempt a zero sized allocation, followed by an arbitrary copyin from userspace into that area
2. You can reach so_bind with an arbitrary addr+addrlen, if sn->domain != AF_UNIX, you copy into the allocation with no size check on the addrlen which is a heap buffer overflow

[slow agate]

we ported dhcpcd to managarm, BPF was one of the things

that was quite quick to implement though

the other major concern was how it configures routes after negotiating dhcp details

which we did with netlink, but idk if dhcpcd has other backends for doing that

the BPF filtering also turned out to be quite useful for udev

[tribal nova]

I wonder if I could do network scheduling better with fancy filtering like APF (Auxiliary Particle Filter) with PSIS (Pareto Smoothed Importance Sampling).

[hexed solstice]

thanks, fixes on the way

[hexed solstice]

i've got netlink and a small subset of rtnetlink

anyway that's dhcpcd compiled, now for the classic "run it, see where something is missing, add it, retry" loop

[hexed solstice]

i sometimes wonder if the devil whispered ideas on how to make it maximally painful to work with from C into the ear of the author of NetLink

someone must have died and gone to hell and taken a copy of the socket cmsg api down and showed it to lucifer himself and asked him whether he could do any worse

[earnest salmon]

lmao

[zenith karma]

surely you could make it work well

e.g. write some kind of static code generation tool for interop

[hexed solstice]

it would make life a lot easier

i am not a huge fan of code gen since it makes the build process more complicated but for IPC it's reasonable enough

the C++ people meanwhile are laughing with their templates

[zenith karma]

aye, laughing all the way into 30 second compile times

[hexed solstice]

great, dhcpcd is sending solicitations:
4 11.191282919 0.0.0.0 255.255.255.255 DHCP 342 DHCP Discover - Transaction ID 0x81ba4b

now to get it handling responses

[sinful jetty]

didnt u say it needed bpf?

[hexed solstice]

it doesn't really

when linux implemented bpf they forget to flush any pending in the receive buffer

so dhcpcd can't trust it anyway and always checks it's getting the sort of packets it wants to deal in

but i have the infrastructure in place to support it for the fun of it anyway

[sinful jetty]

So you patched it out?

[hexed solstice]

no need, it just works if you don't do it

i have it working well enough now to negotiate an address and update route and ip accordingly, but it wants to do something with a DHCP socket involving receipt of a "packet info" cmsg

so now i've got to start working on the whole apparatus of socket control messages

[sinful jetty]

Oh ok

[high solar]

Leo implemented the bpf used by dhcpcd in Managarm, it's really simple

[hexed solstice]

it does sound like a fun activity

i have all the plumbing already so i only need to write an interpreter

[slow agate]

literal 20 minute job

[tribal nova]

I've never looked very closely at BPF. It seemed like a reasonable thing to do. I've just never dug deep into networking.

[hexed solstice]

the next thing i need to do for dhcpcd to be able to run in daemon mode rather than oneshot is to implement flock() and ftruncate()

flock() should be straightforward, ftruncate() is subtle and involves the VMM, so i won't touch it yet as it calls for a clear frame of mind so i can tackle it with a proper awareness of VMM concerns

so i'll do BPF for now instead

[sinful jetty]

what does the script it uses look like

[hexed solstice]

it has a few

for instance here's one of them:

static const struct bpf_insn bpf_bootp_base[] = {
    /* Make sure it's an IPv4 packet. */
    BPF_STMT(BPF_LD + BPF_B + BPF_IND, 0),
    BPF_STMT(BPF_ALU + BPF_AND + BPF_K, 0xf0),
    BPF_JUMP(BPF_JMP + BPF_JEQ + BPF_K, 0x40, 1, 0),
    BPF_STMT(BPF_RET + BPF_K, 0),

    /* Make sure it's a UDP packet. */
    BPF_STMT(BPF_LD + BPF_B + BPF_IND, offsetof(struct ip, ip_p)),
    BPF_JUMP(BPF_JMP + BPF_JEQ + BPF_K, IPPROTO_UDP, 1, 0),
    BPF_STMT(BPF_RET + BPF_K, 0),

    /* Make sure this isn't a fragment. */
    BPF_STMT(BPF_LD + BPF_H + BPF_IND, offsetof(struct ip, ip_off)),
    BPF_JUMP(BPF_JMP + BPF_JSET + BPF_K, 0x1fff, 0, 1),
    BPF_STMT(BPF_RET + BPF_K, 0),

    /* Advance to the UDP header. */
    BPF_STMT(BPF_LD + BPF_B + BPF_IND, 0),
    BPF_STMT(BPF_ALU + BPF_AND + BPF_K, 0x0f),
    BPF_STMT(BPF_ALU + BPF_MUL + BPF_K, 4),
    BPF_STMT(BPF_ALU + BPF_ADD + BPF_X, 0),
    BPF_STMT(BPF_MISC + BPF_TAX, 0),
};

[sinful jetty]

oh its literal bpf

i thought of ebpf

[hexed solstice]

bpf is in and seems to work
bpf_input: filter returned len 4294967295 (msgsize was 590)

[tribal nova]

Excellent news! I need to get to networking at some point soon.

[tribal nova]

I'm not entirely sure what the difference between bpf & eBPF is.

[sinful jetty]

btw did that thing you post attach to some specific function, or how does it work? I was under the impression that bfp relies on specific kernel function names etc

[hexed solstice]

it just operates on the data of input packets: (not my diagram)

[sinful jetty]

ah ok

so its only ebpf that allows attaching to arbitrary functions?

[high solar]

You're probably thinking of bpftrace

which attaches ebpf to kprobes

[sinful jetty]

yep

does mangarm support that?

[high solar]

No

we only have classical bpf, not ebpf

[sinful jetty]

i think ebfp support would be based

[high solar]

ebpf is more complicated, especially on the verifier side

bpf only has registers but ebpf has limited pointer access that needs to be verified before execution

By doing bounds + alignment checks

the verifier needs to track the possible ranges into which pointers can potentially point

There have been many CVEs due to bugs in the verifier

[sinful jetty]

are there any docs for what registers contain what stuff for the classic bpf on entry

i cant find anything since everythign talks about ebpf these days

apparently it assumes BPF_ABS and BPF_IND will just access packet data

[hexed solstice]

just zeroes

https://github.com/sergev/4.4BSD-Lite2/blob/master/usr/src/sys/net/bpf_filter.c#L171 the original interpreter is here

the validator is also trivial

[sinful jetty]

BPF_STMT(BPF_LD + BPF_B + BPF_IND, 0)
So this means load a byte at packet offset 0 + whatever is in the reg (which should be 0 at entry)?

[sinful jetty]

is your code also available?

[sinful jetty]

strange that they didnt do BPF_ABS here

[hexed solstice]

https://gist.github.com/netbsduser/5d95a62264896228d5c48bbc58592789

[sullen fern]

literally dtrace

[tribal nova]

I should set up stuff like dtrace & gdb stubs & such.

[sinful jetty]

cool

[hexed solstice]

i'll take a brief break from the networking stack to start figuring out ftruncate()

i have a rough idea of what ought to be done

[final hatch]

what are you going for?

making it as fast as possible?

[hexed solstice]

i do want speed and scalability, but i want to avoid straying too far from the RFCs also

[final hatch]

which RFCs?

[hexed solstice]

god, there's too many

some i reference are 9293, 5685, 6582, 5681, 4861, 4862

[final hatch]

theyre all internet related. what do they have to do with ftruncate?

[hexed solstice]

i thought you were on about the networking stack

[final hatch]

oh i wasn't

i was going on about ftruncate

[hexed solstice]

the simple goal for ftruncate is for it to work

[final hatch]

my bad

[hexed solstice]

the principal complication is having to unmap pages beyond the truncation point that remain mapped

i had a solution partially figured out

viz. maintaining a list head in a vm_object of all vm_map_entries that map that object

and walking those to clear out the relevant pages

if i can have a barrier beyond which all faults above the truncation point have finished, and thus no busy PTEs are left in processes mapping beyond the truncation point, then i think i can do this relatively straightforwardly

without having to take the mappings rwlock of each vm_map that contains such map entries

[final hatch]

ah damn, I didn't think of that case

but are the pages supposed to go away actually?

[hexed solstice]

unfortunately yes, posix required

a bit surprising as this is the sort of subtlety you would expect posix to be lax about

[final hatch]

i guess for that you need to track all the vm map objects mapping that file in a list and go through that list?

[hexed solstice]

that's my plan, the alternative would appear to be page-level tracking, which i don't want

so introduce that list to the vm_object, link vm_map_entries mapping that object onto that list, guard it with a mutex, take that mutex, and do the unmapping. you would take that mutex whenever mapping/unmapping a vm object

so there exists the ordering vm_map::rwlock (per vm_map mappings tree lock) -> vm_object::map_entry_list_lock

i don't need to hold the vm_map::rwlock to do individual page map/unmap operations though, and that makes life a great deal easier

so the "remove pages mapped beyond truncation point algorithm" is something like
lock vm_object::map_entry_list_lock;
for each map entry, do:
if map entry doesn't extend beyond truncation point, continue
otherwise, do:
for each address beyond truncation point, do:
acquire the needful page pte creation spinlock
if it's valid & points to file data, unmap it
release the taken page pte creation spinlock
unlock vm_object::map_entry_list_lock

in other words there the ordering of interest is vm_object::map_entry_lock -> some pte creation spinlock; and there is no need to hold the vm_map's rwlock

since concurrent regular unmappers will hit that same spinlock as part of the process, and while they hold the vm_map::rwlock, they also have to take the vm_object::map_entry_list_lock after doing their regular unmapping, to remove that map as a map of interest

[hexed solstice]

i've got the concepts of a plan in place

now executing on it

[sinful jetty]

Do you use agents at all during development?

Also how do you approach testing

[hexed solstice]

now that github has these things i asked one to review the vm code and it made some helpful summaries of how codepaths look for me

[hexed solstice]

if i am writing something substantive i write it to run as a userland program first if i can

the current keyronex vmm is the descendent of one of those

[sinful jetty]

Fair

[hexed solstice]

i went to not an extreme but at least a moderate effort to try to make that a workable strategy

so in userland i really did simulate a load of physical memory divided into pages, page tables, and functions to walk and deal in those tables read/write as the tables describe

i find trying to test components like these in the kernel as very difficult so that's the only way i can really do it

and then just try to write some cases which by nature are integration tests

[hexed solstice]

the plan is progressing satisfactorily

[hexed solstice]

the plan continues to progress satisfactorily

i thought i would have to contend with a problem

[obtuse falcon]

i wish i had your discipline

[hexed solstice]

the problem was that as part of the truncation process, you need to follow a list headed in the vnode's vm_object containing every vm_map_entry, and then for each vm_map_entry that extends beyond truncation point, clear out any mappings of a shared vm object page. that's straightforward enough but what if you encounter a busy PTE (one that's created as part of the pagein process?) then you'd have to wait for it to become unbusy, which means you'd have to back out of most of the locks you hold

but in fact you don't! because i added a 'valid length' field to vm_object and made it so that any object page fault, once it's done the I/O and relocked the needful, it checks that length and any pages it brought in beyond the length, it doesn't resolve them to valid PTEs, instead it doesn't map those

so i can just ignore the busy PTEs

the next step is the actual purging of the VM object beyond the truncation point. there i should wait for busy PTEs to avoid an unlikely but possible TOCTOU where valid length is reduced and then increased again meaning that old data was paged in. but that's much easier to do because i'm just dealing with a VM object's page tree, not with vm_maps that map it

so the unlock-then-relock is not onerous at all

[hexed solstice]

it only comes in stops and starts unfortunately

(result of that pleasant surprise: the "deal with mappings above truncation point" is only 100 loc)

[final hatch]

my operating system design kinda diverges from yours in a way, as there is no such concept as a "busy PTE"

instead, when a page fault happens, it will attempt to read from the backing store regardless of whether or not the data was already going to be read in or not

this isn't that great, as two redundant reads are wasteful...

but i saved a bit of time implementing a busy ("transitioning") PTE mechanism

it basically backs out of all VMM locks, reads the page, then relocks the address space lock of the process and sees "oh, there's already a page here, bye bye work"

[hexed solstice]

so for the VM object itslef it may generate a second readin?

in my case i am relying on the busy entry there to properly serialise truncation

the alternative would be implementing some barrier beyond which there's no paging I/O in progress beyond the truncation point

anyway that's truncate support in for 9p and all the VM and viewcache infrastructure, i think

now i can add the syscall

and if all goes well then this diversion from networking stack may have been shorter than i expected

i needed to take this diversion because dhcpcd in non-oneshot mode wants to truncate stuff

but it's certainly good to have finally tackled truncation which i previously imagined would be frightfully painful to actually implement

[hexed solstice]

i need flock as well it seems for dhcpcd

[hexed solstice]

dhcpcd seems to end up with FD 1 being a unix domain socket

i've got little idea why et

it's a socketpaired socket looks like

yes it seems to just work now

[root@keyronex /]$ dhcpcd -4 --oneshot net0
mlibc: syslog: dhcpcd-9.4.1 starting
mlibc: syslog: DUID 00:01:00:01:31:5d:34:d9:52:54:00:12:34:56
mlibc: syslog: dhcp_vendor: No such file or directory (ENOENT)
mlibc: syslog: net0: IAID 00:12:34:56
mlibc: syslog: net0: rebinding lease of 10.0.2.15
mlibc: syslog: net0: probing address 10.0.2.15/24
mlibc: syslog: net0: leased 10.0.2.15 for 86400 seconds
mlibc: syslog: net0: adding route to 10.0.2.0/24
mlibc: syslog: net0: adding default route via 10.0.2.2
mlibc: syslog: exiting due to oneshot
mlibc: syslog: dhcpcd exited
[root@keyronex /]$ ifconfig net0
net0:   flags=41<UP,RUNNING>
        inet 10.0.2.15 netmask ffffff00

so there you have it

[twin haven]

very nice

[high solar]

nice

[tranquil thorn]

very nice

[sinful jetty]

nice

[slow agate]

did you end up implementing rtnetlink or some other interface?

[hexed solstice]

rtnetlink, or at least a subset of it

it's true it was invented by beelzebub in a competition with the authors of the sockets control message API to see who could make the thing the most unpleasant to deal with in C but netlink turned out to be surprisingly a much better fit to STREAMS based async io than the BSD routing socket

it's nice also that it deals not just in routes and updates but in addresses etc as well

[slow agate]

also atomic updates

instead of having to build up state ioctl by ioctl

TLV formats are great for describing this sort of stuff IMO

not that it's impossible with an ioctl, it's just really not that great (see DRM atomic modesetting)

[hexed solstice]

i really disliked that especially with, say you do SIOCSIFADDR followed by SIOCSIFNETMASK, you've got to un- and reinstall any connected route you installed after SIOCSIFADDR after SIOCSIFNETMASK comes in

and for IPv6 it gets complicated to do it by ioctl since there are almost always multiple IPv6 addresses on an interface

[hexed solstice]

i can avoid making the stream head (which is the only place executing in a context which can copy in/out data from userspace) have to know in advance how much data needs to be copied in by that too

there is another way to do "transparent" ioctls (where the stream head doesn't have to know how much data to copyin in advance) with a STREAMS-like system but it's horrible

in the stream head context, responding to an ioctl request, you would have to block awaiting a message from the module handling the ioctl, telling the stream head "copy in this many bytes", which then has to send a message back down with the bytes attached

[hexed solstice]

all that remains for the new network stuff is to integrate TCP into the new order

this may be a little complicated

[slow agate]

good to see the NetlinkHelper being put to use

[hexed solstice]

it really does make life so much easier

[slow agate]

one thing that would be funny is to build an interator for netlink stuff so that you could just use the ranges library

funni but not really necessary

[deft elm]

hmmmm

do you plan to improve this

static tcp_seq_t
tcp_issgen(void)
{
    return ke_time() / 4000;
}

perhaps to something closer to: https://datatracker.ietf.org/doc/html/rfc6528

more recent doc on it: https://datatracker.ietf.org/doc/html/rfc9293#section-3.4.1

[hexed solstice]

i will be doing so in the future

i've seen enough tcp for now but there are many things i would like to improve on there for the future and doing the RFC 9293 should for ISS generation is one of them

for now i'm figuring out how i'll implement PTYs now (i did them before, but devfs and STREAMS are different now so it's time to do them once more)

[hexed solstice]

ptys is back

[tranquil thorn]

nice ui

[hexed solstice]

keyronex is getting a service manager/init system

the activation/dependency semantics will be basically systemd-like. it's worth explaining this in case there's anyone who may find that interesting

we should start by looking at illumos SMF. SMF maintains a directed graph structure in memory whose vertices are:

• services (abstractly speaking; they can be other things because you can have 'delegated restarters' which can manage other resources. there is also a service/instance distinction but that's needless to the explanation)
• dependency groups (e.g. require-all-of, require-one-of, want-one-of, exclude-all-off)
  and edges go from services to dependency groups to services. you can imagine SMF as continuously trying to bring up all services which are not either disabled or in a maintenance state. it determines satisfiability by walking along a service's outgoing edges, checking if the dependency group is satisfied, which recursively checks whether either all services in that group are up, whether one is up, whether all are either up or can't come up (i.e. in a disabled/maintenance state), and whether all can't come up, respectively for each kind of dependency group i mentioned earlier

systemd on the other hand does have a directed graph structure again where edges represent different types of 'dependency'. the term 'dependency' is overloaded in systemd and even individual dependency types combine job-pulling-in behaviours (this will be explained later), post-hoc reactions to state changes, an entirely orthogonal axis of ordering (this is completely distinct from the job-pulling-in axis), and a few other things

it is basically request driven and nothing comes up initially (mostly true! this is not quite the case for devices, since those can't be made to appear by requests) but what is called for by creating a 'transaction' for the start 'job' on the 'unit' default.target

when some job like 'start default.target' is requested the dependency edges of that unit are walked and these generate additional jobs which are attached to a 'transaction'. the most essential feature of a dependency specification of the form "Requires=x.service" is "a start job against x.service must be added to the transaction that activates me, and a dependency link between x.service/start job and me/start job should be created"

note that the job dependency link implies nothing about ordering! ordering only happens with the ordering dependencies Before=/After=. otherwise if b.service only Requires=a.service and you try to start b.service, both are started concurrently and it's a race to see if a.service fails before b.service is already up

anyway, the transaction is filled with all these jobs, and then the job set for a given unit is merged together if possible (e.g. Reload + Restart merges to Restart). many job types are unmergeable, and if these happen, then systemd tries to delete the first one that's not been marked as transitively required in some way by the job carrying out the request the user has made

having proposed the job set the 'transaction is applied' which means replacing any jobs already installed against the affected units with the ones from the transaction. then the jobs can start

as they complete, the before/after edges going out/coming in to a unit are walked to look for other units which has a pending job, and if those units are ready (as determined by their 'Before' dependencies having no job running at this time) they run. if a job fails, then the dependency edge is walked to fail any jobs that depend on it

unfortunately as i've just realised it takes a bit more than 8 discord messages to effectively explain even a simplified model of the principles of systemd's job engine. i'll write a proper document about it some day

[hexed solstice]

i have the core dependency/propagation logic written

so i can get some pretty output:

INFO: Loaded default.target
INFO: Loaded a.service
INFO: Loaded b.service
INFO: Dispatching job a.service:start
Starting Service A...
Started Service A                                 [  OK  ]
INFO: Dispatching job b.service:start
Starting Service B...
Started Service B                                 [  OK  ]
INFO: Dispatching job default.target:start
Starting The Default Target...
Started The Default Target                        [  OK  ]

it doesn't actually do anything yet. i need to design that part carefully!

because one of the key considerations i have to keep in mind is that i follow the SMF approach where the service manager is a discrete planning/scheduling mechanism, which issues requests to actual supervisors, which are asynchronous

(that's clearly the superior architecture; you then get limitless flexibility to dynamically add new supervisors, or indeed simple status-reporters, for any sort of entity you want, and all sorts of entities for which you can write a program that can report the state of them and maybe provide implementations for requests like start/stop/refresh, those entities can then participate in a global graph. it's very modular, very microkernelish)

[sullen fern]

I don't understand much of it but it sounds really cool

init systems are more complex than I thought

[hexed solstice]

https://www.usenix.org/legacy/event/lisa05/tech/full_papers/adams/adams_html/index.html a worthy initial read

[hexed solstice]

i'm writing the basics of the service supervisor state machine today. programs should be executing soon

[hexed solstice]

INFO: Dispatching job a.service:start (id 2)
Starting Service A...
Hello from service A!
DEBUG: Reaped pid 841719 (status 0)
Started    Service A                                 [  OK  ]
INFO: Dispatching job b.service:start (id 3)
Starting Service B...
Hello from service A!
DEBUG: Reaped pid 841723 (status 0)
Started    Service B                                 [  OK  ]
INFO: Dispatching job default.target:start (id 1)
Starting The Default Target...
Started    The Default Target                        [  OK  ]

and that's that

[sullen fern]

thanks!

solaris did a lot of cool stuff

[hexed solstice]

no worries, and this powerpoint is also a nice introduction https://www.filibeto.org/~aduritz/truetrue/solaris10/t-smf-general-march-2005.pdf

[hexed solstice]

this evening's work, adding procdesc support to the service manager

for the sake of portability i am not requiring it but it's nice to have

(i wonder if i should add just keyronex-specific code for procdescs in mlibc, or do it with sysdeps and abi-bits so it could also work on a future mlibc for freebsd?)

(and indeed to any other OS here that chooses to implement procdescs - i recommend it!)

[hexed solstice]

the service manager seems to run OK on keyronex/amd64 but not keyronex/m68k, trying to figure out why. i think pdfork() (fork() but returns a procdesc) is broken on m68k somehow

[high solar]

is a procdesc a file descriptor?

seems that the answer is yes, i see. that makes sense

[hexed solstice]

yes, invented as part of the Capsicum project https://www.usenix.org/legacy/event/sec10/tech/full_papers/Watson.pdf

Process descriptors address these problems in a manner similar to Mach task ports: creating a process with pdfork returns a file descriptor to use for process management tasks, such as monitoring for exit via poll.
When the process descriptor is closed, the process is terminated, providing a user experience consistent with that of monolithic processes: when a user hits Ctrl-C, or the application segfaults, all processes in the logical application terminate.
(the "terminate on procdesc close" is optional if not in capability mode)

init: starting Service and Resource Manager
Starting Plumb net0...
plumbed interface net0
Started    Plumb net0                                           [  OK  ]
Starting Initialisation...
Started    Initialisation                                       [  OK  ]
Starting Bash...
Started    Bash                                                 [  OK  ]
Starting Default Target...
[libc]: mlibc: sysconf(_SC_CHILD_MAX) returns fallback value 25
Started    Default Target                                       [  OK  ]
bash-5.2# srmctl
OBJECT                                   LOAD       STATE           DESCRIPTION
plumb-net0.service                       loaded     online          Plumb net0
init.target                              loaded     online          Initialisation
bash.service                             loaded     online          Bash
default.target                           loaded     online          Default Target
bash-5.2#

there's now enough of the service manager to boot keyronex (not a great challenge given how little there is to do)

for purely aesthetic reasons i think the next thing i will do is implement a trivial getty and login, and a flag for services that tells SRM "when this is running, suppress the initialisation messages sent to console"

SRM has the delegated restarter concept from Illumos SMF where a class of services can be managed by a delegate. conceptually SRM itself just tells delegated restarters when they should bring up, down, or reload a service in the needful order. so a fun exercise will be to write a delegated restarter that receives information about device state

then another delegated restarter, which would handle network management, can be subscribed to those events, and when it finds a NIC has appeared, it can plumb it (NICs can't be used for actual networking without userland 'plumbing' a STREAMS stack terminating into it). the network delegated restarter then gets to decide whether and how it wants to use the NIC. it could decide to configure it with DHCP. and of course the result might be an address object appearing in the SRM graph

[hexed solstice]

init: starting Service and Resource Manager
Starting Plumb net0...
plumbed interface net0
Started    Plumb net0                            [  OK  ]
Starting Initialisation...
Started    Initialisation                        [  OK  ]
Starting Getty...
login: root
[libc]: mlibc: sysconf(_SC_CHILD_MAX) returns fallback value 25
[root@keyronex /]$ srmctl
OBJECT                  STATE           DESCRIPTION
plumb-net0.service      online          Plumb net0
init.target             online          Initialisation
getty.service           online          Getty
default.target          online          Default Target

much more aesthetically pleasing

[sinful jetty]

btw what was the issue with initware that you had to make a new one?

[hexed solstice]

initware is a systemd fork and turning it into what i'd want it to be is more work than writing my own

[sinful jetty]

Ah

What are the fundamental differences

[hexed solstice]

SRM conceptually is intended to be a lot like Illumos SMF only with systemd's job concept and methodology for ordering, requirement and propagation of events

and SMF's most distinctive feature, i would say, is that it allows things to participate in a global graph made up not only of services but also of any kind of entity for which you can write a 'delegated restarter' that can report on the status of the entity being supervised and perhaps answer requests to 'bring up' or 'bring down' that entity

[sinful jetty]

Interesting

[hexed solstice]

but the biggest difference that's straightforwardly visible is that SMF is an order of magnitude superior in engineering, observability, and 'explainability'

they really went to extremes in reliability, the master restarter svc.startd can crash and it will be restarted and reconstitute state

[sinful jetty]

Who will it be restarted by if it crashes?

[hexed solstice]

by init (which if it exits would, in turn, be restarted by the kernel)

[sinful jetty]

So it has a restarter mechanism duplicated in multiple places?

[hexed solstice]

there's a librestart for sharing logic between the delegated restarters but since init is trivial, it just does its own thing

[sinful jetty]

Ah

[hexed solstice]

i'm souring on the systemd activation/dependencies model

it's too much of a confused mess

for example, first prize to anyone who can tell me off the top of their head (i know how, it's not very pretty) how to express this relationship between service x and y: x needs y to be up before it comes up; if y then goes down, x also should go down; if y goes back up, so should x

[flint anvil]

it's just a requirement, and the second server should always run?

[hexed solstice]

• before x can come up, y must be up
• in the event that y goes down, x must go down too
• if y then goes back up, x should go back up too

[flint anvil]

I don't get the question

It's a bidirectional relationship

What's probably more complicated is when you start getting cycles

[hexed solstice]

the question is, who can tell me how to express this simple relationship in systemd unitfiles, without studying the systemd source code?

[flint anvil]

oh

sorry, I misread the whole thing and it confused me

[hexed solstice]

i've got the concepts of a plan of what i should do

[flint anvil]

I have a cursed thing, which looks at stuff published on pmbus (mbus clone ), and starts the services as the objects are published, so it creates arbitrarely long chains of services (drivers) that get started as objects (devices, other services, etc.) appear in the system. And those die off naturaly if devices/servers disappear, which deletes the rights to those objects and makes everything exit

[zenith karma]

how do you define when to start?

and have you just reimplemented upstart with extra steps

[flint anvil]

nice

[flint anvil]

with yaml

[hexed solstice]

ubuntu's old thing, now used by chromeos https://github.com/ubports/upstart/tree/xenial

[flint anvil]

yeah, the wikipedia was the second link for me

it's just very sad how bad search engines are getting all the time

[hexed solstice]

it had an inverted model

instead of x needs y it's 'x starts on y starts'

[flint anvil]

with everything returning some startup seo weirdness instead of long established projects

[hexed solstice]

it's mostly of historic interest now in fairness

you can see in it certain elements prefiguring systemd (which was called BabyKit at the time, don't ask me why)

[hexed solstice]

my thinking right now is to basically use a slight variation of the SMF activation model

namely, every enabled service wants to be up and the service manager enthusiastically tries to bring them up as soon as their dependencies are satisfied

i make a slight change and replace SMF's dependency-groups (used in SMF for a "require any of" dependency) with 1:1 dependencies of type require, optional, exclude + a virtual service concept

require = dependency must be up; optional = dependency must be either online or it is (transitively) impossible for it to come up; exclude = dependency must be (transitively) impossible for it to come up. and then for propagation purposes there is another property of dependency edges, whether error-caused failures, ordinary restarts, configuration refreshes are propagated such that if the dependency goes down/is refreshed, the dependent also goes down/gets refreshed

and virtual services are special nodes, which satisfy those tests of dependers with the best result from testing each of the services that provide the virtual service

this starting point makes for a very easy to reason about system, with the exception of virtual services. maybe it's fair to say that the configuration tool should only permit one provider for a given virtual to be enabled, and all the other providers of that virtual must be disabled. that turns them into aliases

that makes it easier if we want to consider an approach where we relax the principle that every enabled service wants to be up, and instead have a principle of demand, where some set of services are picked by the administrator as services that he wants to be up, while services unmarked go up only if they are demanded by services that are marked 'wants to be up'

[hexed solstice]

i'm working on the new service manager

the basic division of labour: srmd is the principal supervisor of services and maintains the graph structure of services and other entities (which may be externally supervised or observed) exposed in the service graph; cfgd maintains service configuration (there are three 'layers' of which; service definition files, customisation files, and command-line/programmatically-set configuration changes; provenance tracked, so something like 'blame' should be possible to see why some property has some value in some service); and srmcfg is a tool that parses configuration files, communicates with the interface cfgd exports to install them, and allows for configuration changes

[hexed solstice]

i'm using varlink for the proof of concept. god if it isn't ill-conceived

i'll do doors some day

[hexed solstice]

current priorities:

• the service management suite/"init" system
• bring back the framebuffer console but this time with a sidebar including a classy analogue clock

[neat steppe]

your sidebar's not authentic if its not drawn by the system firmware!

[hexed solstice]

but if it has no analogue clock it has no class

[sullen fern]

a nice goal would be putting the fb console in userspace

[final hatch]

i did that

[hexed solstice]

i think this is very reasonable

the kernel can keep a trivial one as a debug console

[hexed solstice]

logic for propagating state changes through the service graph gets a bit ad-hoc the more problems you deal with

suppose X requires Y requires Z. each require edge has full stop propagation enabled. and X, Y, Z were up. then Z goes down.
for each edge to to Z, if it's requires, i ask the service it comes from to stop. so Y goes down next, and when that goes down, in turn i find the require edge from X to Y and ask X to go down. but for explicit service disablement i mark the subgraph transitively requiring Z as "to go offline" and for each leaf i bring it down, and propagate in the opposite direction, so X goes down first, then Y, then Z. whether that same principle should be applied for the post-hoc "Y stopped so stop its dependents" is very debatable. because the proper ordering has already been violated! Z is already down and it should've been the last to go down!

(the X Y Z case also already illustrates how it's easy to bring things up, people have made the point for years that parallel startup could be done with a Makefile, but that's trifling. the real question is how do you deal with the fact that you are not running a batch job but trying to maintain a dynamic system?)

it's worth talking here about what systemd does here as a comparison: if X Requires=/After= Y and Y same for Z, then an initial failure of starting Z would cause Y not to come up, but once Z is up, Y can come up, and once Y's start job is completed, that's the end of the story. Y is up and won't be brought down by Z failing. the systemd model is fundamentally imperative. so if you want Y to go down if Z does, then you need to use BindsTo= which will cause a stop job to be queued for Y if Z goes down. but then if Z goes back up, Y remains down; you can cause Y to go back up, if you put Wants=Y into Z's definition!

and i think that nicely illustrates that full compatibility between two service managers requires them to be basically the same. the full behaviour is very rarely publicly specified anyway, and if it is, then you have probably conditioned particular implementation details

but i digress. first i want to say that dependency types like "optional" have an interesting implication in that they mean you need pay not only attention to whether a service is satisfied, but also to whether it is unsatisfiable

suppose X optional Y requires Z and none are up yet. Z fails to go up and moves into a failed state. you can't just reevaluate Y; Y is not in a failed state itself, but its dependencies are now unsatisfiable because Z is in a failed state. that means in turn that, since Y's dependencies are unsatisfiable, X ought to go up, because it is no longer waiting for Y to go up, Y isn't going to go up

[neat steppe]

is there any analogy to be made here with the device tree in a plug n play io system

[hexed solstice]

so rather than baking in logic each time i want to handle a case like this i've been thinking of an epoch-based scheme

i have a vague notion of counters in service nodes and having dependency edges track these with a 'last acknowledged' counter

[hexed solstice]

i think there is, they are both cases where dependency graphs which have to take account of a dynamic system rather than a batch one

i'll try to contrive an example, if i have a filesystem mounted on a software RAID which is using some USB drive and someone pulls the USB drive, you have a problem, the proper teardown order has been violated

and likewise there it's debatable whether you should then try to teardown in orderly fashion the FS mount then the software RAID instance

and if your drivers want to try to handle this case and possibly even recover themselves if the USB drive comes back then you really make it interesting

if you talk about the problem abstractly with devices X, Y, and Z then it becomes harder to say what the "right" approach is, just like it's hard to talk about this abstractly with services X, Y, and Z. (i have dependency edges carry both a requirement/optional/exclusion mode as well as a propagation mode of none/on error/on restart to try to offer some flexibility in how this is approached)

[neat steppe]

the raid example is a good one because the guy who designed iokit (godfrey van der linden) gave a wwdc presentation about it in like 2003 where he specifically mentioned the software raid driver as a pain in the ass to write because of tear-down

he was like "we've come up with pretty good patterns for tear-down for you guys to use in your drivers, sometimes these patterns fail but lucky for you you probably wont have to write a software RAID driver like we did"

https://github.com/apple-opensource/AppleRAID

thats it btw

[hexed solstice]

if you find this again, post here

[hexed solstice]

i'm really inclined to go with a model where nodes store epoch numbers and dependency edges store an observed epoch and get rid of all the ad-hoc propagation code and instead do it centrally

i think this makes a considerably cleaner approach

[deft elm]

software.... raid... ew

one of those things where hardware can do a much better job

[neat steppe]

whats ur reasoning

[deft elm]

because it can handle failure modes pure software can't, like a physical power cut to the device... for example my raid has a capacitor backup, and you can get them with a built in battery too.

also one less task the cpu has to do

[hexed solstice]

i've been moving ahead with the approach to the service graph i described yesterday

[hexed solstice]

interesting point about this: if X requires Y requires Z, and each has refresh-propagation enabled, and your refresh Z, you would generally want first Z to refresh, then after it's refreshed, Y, then finally X to refresh

but what if one of them doesn't implement a refresh method? then usually you would want to fall back to restart

[hexed solstice]

i'm getting totally paralysed with questions like this

time to just pick semi-arbitrary answers to each question and thus implement it, the service manager cannot be all things to all men

[hexed solstice]

so the new concept of a dependency edge is that it's one of require/optional/exclude, and propagation properties: follow down = if dependency goes intentionally down for any reason, I go down; on_maintenance = if dependency goes into maintenance, I go offine; on_restart = if dependency is explicitly restarted, I restart after him; on_refresh = if dependency is refreshed, I refresh after him

i'm not fixed on these and i think distinguishing an intentional down (disabled, or maintained) from an incidental one is useful, but if X requires Y and Y crashes and autorestarts it's often the case you don't want X to react to this

[hexed solstice]

i've distinguished target down propagation from state down propagation. you can declare X requires/restart_on=state-down Y (same) Z. an interesting effect is that the stop ordering logic, if Z goes down unexpectedly, will cause X to go down before Y in this case; and then if Z goes back up quick enough, only X will restart, Y won't. i'm not fussed about this, it should all still be simple enough to document

[hexed solstice]

i'm revisiting turnstiles to figure out how i'll support priority inheritance with userland locks

(a goal being they don't need to enter the kernel in the uncontended cases)

[hexed solstice]

i think i can do this a bit more straightforwardly!

i have an existence guarantee on all thread structs within the turnstile blocking and priority-willing path because they're RCU-freed

and the turnstile chain heads are permanent. so i can stop thinking about lifetime altogether and that enables a lot of relaxing

[hexed solstice]

i think i now know how to implement a userland mutex that participates in turnstile priority inheritance

i stole most of the concept from linux's PI-futex and ROBUST-futex

[neat steppe]

i mean its pretty simple isnt it

dont you just CAS the userspace lock word the same as you would with a kernel space one

(w/ all necessary validation etc etc)

[hexed solstice]

it's conceptually similar to yours but has to be resilient against several conditions

[neat steppe]

but the userspace lock word contains a thread ID

and youd handle this in a callback and abstract away the turnstile package's knowledge of the lock word

[hexed solstice]

like the owner died; someone trampled it; etc

and interruptibility is a must

then you need to add additional protections

[sullen fern]

couldn't that happen with a kernel lock as well?

[hexed solstice]

provide a way for userland to lightweightly register the lock it is trying to/has acquired, and at thread exit, consult that and do the needful transitions. but this is not guaranteed, so you need to handle the case they didn't do so

[hexed solstice]

if an owner dies while holding a kernel lock then you may as well panic, likewise if it was trampled

while interruptibility and timeouts are only nice to have for kernel locks

so i have cause to add code to walk the pi list of a thread at exit and unlink the thread from any chains it's still on

[hexed solstice]

keyronex is getting a hypervisor

for RISC-V only for now

[high solar]

very cool

[steep marlin]

Nice. Don't think many hobby OSes have this, do they?

[sinful jetty]

Cool, why riscv of all things?

[steep marlin]

He was talking about the H privileged extension yesterday so I'm guessing that

[sinful jetty]

is there even riscv hw supporting that?

[steep marlin]

I recall something with that being either super new or on the horizon

[hexed solstice]

i remembered bonzini's submission to some riscv mailing list and that they'd basically adopted his proposal wholesale

[sinful jetty]

i wish they listened to actual professionals for other aspects of the arch as well

[hexed solstice]

i first knew of bonzini because he maintained gnu smalltalk for a while and later he went on to KVM/QEMU. very talented man. it's really quite pleasant to deal with RISC-V virtualisation as a result

if they had gone ahead with their original proposal, they had basically copied AArch64 as it was prior to the addition of VHE (which btw had been 2 years before they made their original proposal) so they would've ended up adding something equivalent eventually as yet another extension and we would have another degree of freedom to contend with

[sinful jetty]

btw it's only possible to test with tcg qemu now right?

[hexed solstice]

yeah, unless/until H-mode boards are available

@high solar kindly shared a link to one, and i will have a look for information about it

[sinful jetty]

looks like the minimal spec one is 300 dollars

[hexed solstice]

~ # busybox uname -a
Linux (none) 7.0.8 #9 PREEMPT Sat May 16 00:46:55 BST 2026 riscv64 GNU/Linux

a hypervisor for RISC-V capable of booting a minimal linux with initramfs turns out to be an insanely simple undertaking

[steep marlin]

that's nice

[sinful jetty]

can you show the code

[hexed solstice]

it's on the way with the risc-v port

[obtuse falcon]

now there is

[sinful jetty]

Id like to get one but 300 dollars for the base model

[obtuse falcon]

F

ram is expensive

[sinful jetty]

The base one has like barely any ram

[sinful jetty]

Do u have access to it already?

[obtuse falcon]

no lol

[sinful jetty]

Oh

[high solar]

It's arriving in ~1-2 weeks

[high solar]

Well, they do?

What aspects do you mean

[sinful jetty]

the page table format

[sinful jetty]

damn

another ci target?

[sinful jetty]

cpu feature discovery

[high solar]

Both work as intended

[sinful jetty]

why wouldnt they work as intended

or how could they even NOT work as intended

you can make the worst design possible it would still work

[high solar]

I don't think anybody is complaining outside of x86 centric hobbyist space

[sinful jetty]

do u have insight into conversations between people implementing it for production kernels?

[high solar]

If you're asking about rich OS kernels, linux is by far the biggest consumer

But afaict it's well perceived both in embedded and rich OS

[neat steppe]

it's problematic for anybody who uses recursive page tables which work easily on everything except riscv

[obtuse falcon]

it's just not a big deal

[neat steppe]

It causes them to have to spend more memory doing an explicit linear mapping of the page tables and change some code bc changing a PDE no longer magically causes the new page table to appear in the correct location in the recursive mapping

[sinful jetty]

there were some other features people brought up, I just dont remember them

[obtuse falcon]

it's a different platform, imo it's expected to be different, for better or worse

[sinful jetty]

but good virt extensions is a win ofc

[neat steppe]

Recursive mapping is not an x86ism

[obtuse falcon]

didn't say it is

[neat steppe]

It's an everything but riscv ism

[sinful jetty]

its just that it was supposed to be a perfect arch developed after decades of experience with legacy architectures

completely free of legacy cruft

[obtuse falcon]

that's impossible

[sinful jetty]

which part

[obtuse falcon]

no legacy

[sinful jetty]

what legacy does risc-v have?

risc-iv?

[obtuse falcon]

sbi calls, csrs like misa, and i think whatever interrupt controllers came before AIA

[sinful jetty]

sbi is like the best part wdym

[obtuse falcon]

no i mean some

[sinful jetty]

oh

[high solar]

The parts that ppl actually complain about for production are different from the parts that hobby projects complain about

[sinful jetty]

what do they complain about?

[obtuse falcon]

the ones replaced by sPI for example

[high solar]

see Svrsw60t59b

[high solar]

yeah, or that

[obtuse falcon]

my biggest pain point with risc-v is not an isa issue, but just how hardware implements it

[high solar]

Native replacement for SBI stuff etc

[lethal osprey]

most of my issues with risc-v are that the "base" versions of a lot of things are too minimal

[obtuse falcon]

wdym

[high solar]

That's why profiles exist

The base versions are not meant to be useable as is

[sinful jetty]

whats it about?

[high solar]

Also, the lack of cpuid is intended

[sinful jetty]

i mean Svrsw60t59b

[high solar]

More AVL bits in PTEs

[sinful jetty]

ah

[lethal osprey]

for example "base" SBI (without any extensions, including the legacy ones) does literally nothing useful

[obtuse falcon]

yeah but that's the whole idea

you just don't target the base for general purpose systems

like, an rv64im cpu doesn't make sense to exist in an SBC

[high solar]

the general purpose cpu is supposed to be rva23 (or another rva*)

any of the A (= application) profiles

[obtuse falcon]

yeah rva23 is a good baseline - minus the downside of having one implementation in hw at the moment

[high solar]

In contrast to the embedded profiles

[sinful jetty]

what profile is that 300 dollar board that just released?

[hexed solstice]

great, i look forward to hearing about whether it's any good

[hexed solstice]

that was SPUR

RISC-III was SOAR

SOAR was designed for running Smalltalk

[sinful jetty]

lol

[hexed solstice]

apparently it included an iAPX 432 mode too

[obtuse falcon]

RVA23

[sinful jetty]

does that have like everything you would expect as in atomics etc?

[hexed solstice]

i think it's quite robust and has everything important, even H extension

[sinful jetty]

cool

[flint anvil]

Does it have ACPI?

[high solar]

No, i think the milkv titan is the only acpi board so far and it is questionable whether it'll become generally available or not

[final hatch]

arm took a few generations of MMU hardware to learn this lesson

only in armv7 did they implement a proper page table layout which can be made recursive

[high solar]

Is there any OS doing recursive mapping even on current ARM?

[neat steppe]

NT

[high solar]

Does it do it on arm? Did anybody confirm that by decompiling?

Because NT can also use a direct mapping AFAIK and i would be surprised if they were not unconditionally using that on arm

Since arm has a lot stricter requirements around invalidation than x86

[neat steppe]

Well that's not the reason they have it on 64 bit still

It's because of PTE address math

Their mm has a deeply engrained notion of being able to do the PTE address math

[neat steppe]

They've always used a direct map to access remote process PTEs (when possible, for example the low 256MB of phys mem on 32 bit x86 was direct mapped for this I think; when not possible they would map it into a system PTE temporarily)

And a recursive mapping with fancy PTE address math to access local PTEs

[sinful jetty]

Does nt even have a risc-v port? how is that even possible if they dont have a coff for it

[flint anvil]

You can emulate it

(the recursive map I mean)

The coff spec would probably just be from MIPS or whatever?

[high solar]

Yeah you don't need the recursive map for the pte math

As i said, I'd be surprised if windows actually uses recursive paging on ARM

Stuff like break before make are probably way more painful with recursive maps

And also the fact that ARM has separate pts for positive and negative addresses

[sinful jetty]

why was that needed?

[final hatch]

Why wouldn't they?

Why is ARM special

I mean armv6 and earlier didnt have page maps conducive to recursive mappings, but I don't think they got NT working on those anyway

armv7 and aarch64 are the only ones I know of that ran NT

[high solar]

On arm you cannot change the pa of a mapping w/o unmapping it first and invalidating because the cores may share tlb walk caches

[high solar]

Because a lot of tlb state that's allowed on x86 is instant ub on arm

[sinful jetty]

Can you elaborate? What's the problem that may arise if I replace the pa and invalidate tlb right after?

[high solar]

UB

[sinful jetty]

can you elaborate

[hexed solstice]

I was slightly surprised by it too but I'm fairly sure someone provided clear evidence of this

I can't remember who or where

[high solar]

For, example, the cpu may use the old pa or it may use the new pa but old permission bits

[rain flame]

Basically anything that the arm spec marks as ub means an impl can literally do whatever it wants in that scenario

[sinful jetty]

yeah but like how is that even possible if I invalidate TLB

are u saying tlb invalidation doesnt invalidate tlb

[sinful jetty]

no i mean like

whats the underlying problem requiring clearing the pte first

[rain flame]

Oh I see what you mean

[hexed solstice]

Suppose you have mapped a file cow in your app

[high solar]

Gemini also claims that windows does not use recursive paging on ARM

[hexed solstice]

The app tries to write to some of it and your page fault handler runs because the shared page is mapped readonly. Pf handler ultimately allocates a page and copies the contents of the shared file page into it, and maps it as your new private page writeably

[high solar]

And that there is a function MiFillPteHierarchy that does the flat mapping w/o recursion

[sinful jetty]

right, but why isnt just one invalidation enough in this case

[high solar]

Because CPUs may share tlb walk infrastructure

[sinful jetty]

i dont understand why that matters

[hexed solstice]

This guy seems to say it's using a recursive mapping https://connormcgarr.github.io/arm64-windows-internals-basics/ and I thought this was where I learnt it from but on a very quick skim I don't actually see a smoking gun

[flint anvil]

Isn't this a problem on x86/other architectures as well?

[sinful jetty]

Apparently on arm you need to zero the pte first

For some reason

[high solar]

Assume you have two cpus and they share a tlb. W/o BBM, there could be a state where cpu 0 fetches the old pte and cpu 1 fetches the new tlb (both potentially speculatively). If that happens arm allows the hw to do whatever (i.e. take some bits from one pte and some bits from another or do random garbage)

[high solar]

You need to clear the present bit, yeah

[sinful jetty]

Why does that matter if I do a proper tlb flush on all cpus after updating it?

[high solar]

You don't necessarily need to zero it

[high solar]

Because the flush it not atomic among different cpus

[sinful jetty]

If the tlb is shared, why would they refetch the same pte?

[hexed solstice]

There is the possibility that another core effectively acts with, for example, old physical page and new write permission

[sinful jetty]

so like, u have to do something like this?

pte->present = false;
broadcast_flush(va, mask);
update_pte(pte, desired_stuff)

[high solar]

yes

[sinful jetty]

technically cant u do it in this order on x86 also?

[high solar]

yes

But you may be doing unnecessary invalidations if you always do it on x86

For example when going from a less permissive to a more permissive page

You don't need to invalidate on x86

[sinful jetty]

right

[high solar]

Unless you want all cpus to see the update immediately (although in the common case you do want that)

[sinful jetty]

i mean if u dont broadcast they just have to take an extra no-op pf

maybe if the range is large its worth it

[somber brook]

I can't imagine how Gemini could have obtained accurate information, and I don't know where it got this from, but the information is totally wrong.

Both ARM64 and AMD64 use the self map PXE.

[high solar]

i'm not claiming it's correct, could be plausble sounding nonsense

what's self map PXE?

[somber brook]

Yes, that was quite a serious problem, which is why a new facility called Micro-Executive was created to address situations where kernel PTEs need to be mapped as non-global.

[somber brook]

The Windows memory manager maps the page table hierarchy of the current process into the KVAS using a self map. By pointing self map PXE entry in the top level page table to the top level page table, the memory manager creates virtual addresses that can be used to access the entire page table hierarchy.

[high solar]

how does that work?

[high solar]

Is the Keyronex RISC-V virtualization work already pushed somewhere? We probably also want to look at RV virtualization soonish and I'm wondering what kind of changes that'll involve

[hexed solstice]

Not yet I've made a sow's ear of my local branch

But I didn't find it at all invasive to implement the very elementary stuff (there is not even any mmio emulation- just providing SBI) to get a Linux image runnable

[hexed solstice]

i think i'm going to redo pmap in terms of C++ templates

[tranquil thorn]

based

[hexed solstice]

i had been thinking about this already before because some arches call for, ideally, a runtime choice of a pte format, page size, etc. on m68k, 68040/060 MMU is almost a subset of the 68020/030 MMU, but not quite. you could use conditionals but that just isn't very pleasant nor performant. while with C++ templates you can have the whole "create PTE cursor" operation monomorphised

for virtualisation it becomes even more pressing since i want to be able to have two specialisations of "create PTE cursor" available at once

[hexed solstice]

i'm going to make most of the virtual memory system generic on operating on host v.s. guest page tables