Jeff BezOS

[nova fiber]

actually i could probably also use clangs rtsan for this at runtime

[nova fiber]

i hate my life

that'll teach me to allocate guard pages

[nova fiber]

for my next bug its swapgs related yet again

i think my scheduling interrupt is happening during a syscall so im in ring0 but gs hasnt been swapped yet

which causes issues

honestly it might just be easiest to store the kernel gs value on the top of every cores ist stack and then use that

[nova fiber]

yeah this check is getting hit

attaching gdb slows down qemu sufficiently that it doesnt happen

the same check also fires in the interrupt dispatch routine

so im guessing it goes something along the lines of ```

• syscall happens
• interrupt fires before first instruction of syscall is run
• swapgs doesnt happen in the isr dispatch because context->cs & 0b11 == 0
• kaboom

[trim owl]

you can make the cpu clear RFLAGS.IF on syscall

via the mask msr

ah this was already answered in #osdev-misc-0

[nova fiber]

it would appear all of my crashes were because i forgot to add the stack size to the stack base when allocating it while doing smp startup

i may have brain damage

[nova fiber]

a bit delayed on multiple processes, my page table manager was built to only handle one address space, so now i've got to fix that and split it into a kernel and process page table manager

[nova fiber]

fun bug of the day, i trigger ubsan on a misaligned load but only when using the ovmf qemu bios, seabios doesnt trigger it

[nova fiber]

multiple processes work

now to go and do elf loading so that you can actually launch them in userspace

also added the required patches to llvm to cross compile i think

I guess im also getting to the point of needing to add a libc for the userspace

[nova fiber]

there are also a few more touch ups for my packaging tool needed

right now every package phase runs before the next phase so i cant have a build step depend on a configured result from something else

so for pkg-config i need configures of some packages to depend on the installed pkg-config

[nova fiber]

bash source code is so fucked

also what is freedesktops obsession with glib

why does pkg-config need glib

compiling bash has made clang shit itself

[nova fiber]

added progress indicators because i got worried while llvm was extracting

they're buggy as shit because i use a really bad library and i think i also use them wrong but oh well

i guess i need to get the chroot and mount stuff working tomorrow so i can run pkg-config and cross compile stuff with my libc

which i have named jlibc

because

jeff

libc

[nova fiber]

other random things, i quite like the NT thing where every handle has an implicit mutex you can wait for so i think i'll be doing that over implementing flock and equivalents, the linux clone api is pretty good, and linux namespaces are also good but i do wish there was an easy way to get a more complete chroot setup so maybe i'll look at doing something like that

[nova fiber]

cross compiled libstdc++ in the most goofy way possible but it works

i should probably break my reliance on gcc libstdc++ internals for my kernel

i only rely on stuff accidentally because they forgot to mark tons of stuff as not available in freestanding environments

i also copy pasted their entire chrono header verbatim as well but that should've been freestanding to begin with :p

[nova fiber]

great start to the day

at least gives me the opportunity to make sure that i can build the project from scratch

it does not

accidentally gitignored a meson wrap

but aside from that it does build with just source ./setup.sh and ./qemu.sh

[nova fiber]

close enough

[nova fiber]

bash now configures when passed jlibc

doesnt build but one thing at a time

[nova fiber]

gonna be honest i thought bash was more self contained

[nova fiber]

got slightly broken subprocesses working

the init program now launches the shell as a seperate process

[nova fiber]

I think im going to spend some time going back and testing bits that i dont feel so sure about

and untangling some of the more questionable decisions i made

[nova fiber]

I have found rpmalloc which appears to be a reentrant malloc

if it is then i should be able to use this as my interrupt allocator

right now the only thing stopping me from being entirely preemptible is that the scheduler allocates memory, so i have to disable interrupts when fetching or adding work items to the scheduler

[stone igloo]

dont allocate in the scheduler lol

thats a recipe for disaster

[nova fiber]

my scheduler is a moodycamel concurrent queue right now

[stone igloo]

holy fuck what

thats not even super fast

because while the queue is fast

[nova fiber]

i will make it not that at the point i have fixed the rest of my code

[stone igloo]

what you want is (a) more latency than throughput

and (b) latency kinda doesnt matter if you do it on a 1ms timer

and (c) locality is way more important

[nova fiber]

i also dont do any timer training so i have no clue what the scheduler interrupt is running at

[stone igloo]

lmfao what

[nova fiber]

i know its horrible, its just not been something i've cared to fix yet

i will eventually figure out the interrupt speed and use multiple queues and preallocate so i dont waste time, etc

[calm shell]

scheduler cannot depend on the memory manager

bc the memory manager has a much harder dependency on the scheduler later

[nova fiber]

right now i store all the scheduler memory in a fixed size block thats never paged out

its got a seperate pool from the rest of the OS

[calm shell]

what do you need to allocate exactly

i feel like you can move any allocations to thread creation time

do them ahead of time and move the failure case there

[stone igloo]

he's using a third party lockfree queue

which is fast

[calm shell]

ik but i dont know how the data structure works so im just asking

[stone igloo]

at the cost of allocating

[calm shell]

what it allocates

[stone igloo]

probably linked list nodes

[nova fiber]

its basically a deque internally

[stone igloo]

and its hard to make a third party queue behave correctly

[nova fiber]

it gracefully handles oom at least

[calm shell]

if so then you can definitely allocate that at thread creation time

[stone igloo]

do you have exceptions?

[calm shell]

not necessarily as part of the thread itself

[stone igloo]

assuming you are the queue implementer which isnt the case here

[calm shell]

just make sure M>=N at all times where M is number of linked list nodes available and N is thread count

[nova fiber]

I can modify the queue code as much as i need to

[stone igloo]

actually true

[nova fiber]

no it uses new (std::nothrow)

[calm shell]

another example of a thing allocated and destroyed alongside threads but whose lifetime isnt tied to that of the thread it was allocated alongside, is solaris turnstiles

they just need turnstile count >= thread count at all times and threads like juggle turnstiles around and whatever turnstile is attached to you when you get to the termination code is what gets freed

[stone igloo]

oh damn thats cool

[nova fiber]

placement new and that new overload are what make most of modern C++ usable without exceptions

in embedded environments anyway

[zealous mirage]

wow, they added a new that's fallible without exceptions

that could be quite useful

[stone igloo]

that's quite old i think

[calm shell]

isnt that just malloc at that point

[nova fiber]

hm neat, i will have a look at that

[nova fiber]

c++11 iirc

[stone igloo]

old

[nova fiber]

oh wait thats more than a decade ago

[stone igloo]

yeah

lol

[zealous mirage]

it also runs the constructor

[stone igloo]

no

[calm shell]

true

[stone igloo]

because it is not undefined behavior

and malloc is

because c++ moment

(you can use placement new manually too and thats also not ub)

[nova fiber]

malloc is no longer UB in C++20 for trivial types

[stone igloo]

true

[urban schooner]

Malloc automatically starts the lifetime for PODs I think

[stone igloo]

yeah since 20

and nothrow is from 11

[nova fiber]

anyway placement new and std::nothrow are great, big fan

[zealous mirage]

i think with magazines there is actually a good case for a thoroughgoing decoupling of construction/destruction from allocation/deallocation

since the principle of magazines is that you have constructed objects in the magazine

but i suppose this isn't really compatible with how people tend to write C++

[stone igloo]

hmm

how does nothrow work, exactly

[zealous mirage]

they use references and all kinds of things which i understand have to be initialised by the constructor

[stone igloo]

you can use placement new as well

[stone igloo]

is it just compiler magic?

[zealous mirage]

you'd have to adopt a discipline of having the constructor only do whatever initialisation is particular to the kind of object rather than to the particular parameters it's instantiated with

[zealous mirage]

but then you have to construct every time

it's debatable what the right thing is to do really

[nova fiber]

it calls an overload of operator new that returns null on oom rather than throws bad_alloc

[stone igloo]

the answer is no

yeah

[nova fiber]

operator new is allowed to return null, its just that the default implementations all throw

[stone igloo]

struct Lol {};
void* operator new(unsigned long l, Lol p) noexcept;
void p() {
    new (Lol{}) int();
}```

this is fine

[nova fiber]

overloading operator new lets you do very strange things

[stone igloo]

which means you can do new (magazine) T();

for any magazine type

[nova fiber]

you can also overload operator delete for extra fun mismatched new/delete bugs

[stone igloo]

lmao

i'd just use magazine.free for that

or something

[zealous mirage]

for the benefit of @nova fiber to know wtf i am talking about: in the slab allocator with magazines, you can associate a particular allocator (one that exclusively allocates objects of a particular type) with a constructor and destructor function. the magazines are small CPU-local caches of pre-initialised objects that can be popped and already have generic initialisation done and then further intialisation is another specific function. for example you might have an object that contains a condvar and mutex and the objects on the magazines will have had a constructor that initialises the condvar and mutex already called on them. when they're freed back to the magazine, the condvar and mutex remain initialised (hopefully you didn't leave the mutex locked or you're dead) so that future allocations, if served from that cached object, don't need to reinitialise those members

[nova fiber]

oh so basically an object pool

neat

[zealous mirage]

exactly that

it's easier in C since construction is informal but i think would be more challenging with C++ because idiomatically you are using references and such which need to be initialised at construction time

and if you wanted to introduce a distinction between common construction logic and the rest of the construction, you can't use a constructor anymore and potentially not the reference fields you want etc

[stone igloo]

you could use a constructor, you just need to mess around with this custom operator new mechanism

[zealous mirage]

however for objects whose constructors don't require additional parameters that will vary depending on what you want to do with the object, it's very doable

[nova fiber]

putting references in objects is generally considered bad style anyway

as long as everything has a default constructor you should be fine

[zealous mirage]

i think for a lot of things you could get away with it no bother

while if the constructor initialises lots of things based on parameters that vary widely (as opposed to a few fixed combinations of what it's initialised with), it'll probably be less useful to try to cache these at all, since most of the object will be initialised very differently by different uses

[olive musk]

for this I just did C-style constructors, even in C++

[nova fiber]

wow doing unmapping correctly for 2m pages is a real bitch

oom handling makes it much more complicated

[sour geyser]

Maybe I'm missing something, but what do you need oom handling for in unmapping code?

[nova fiber]

if you map a 2m page and then unmap a 4k element in the middle of it you need to then allocate a new page table to store the 4k mappings

and it gets annoying when the unmap request crosses the ends of 2 2m pages so you end up needing to allocate the tables ahead of time otherwise you can oom midway through unmapping and end up with inconsistent page state

0023 and all my tests are passing at least

i'll need to write explicit test cases for the oom behaviour tomorrow

but if i did everything right i should be able to survive oom and not corrupt mappings

150 lines for mapping pages and 350 lines for unmapping pages

resource deallocation is somehow always the more complicated bit

after checking the win32 docs again i now realize that VirtualAlloc doesnt even allow partial unmaps, only posix munmap does

well shit

well it works so whatever

Actually tbf I do use partial unmapping already and I like the pattern since it makes pre allocating sparse date easy

[sour geyser]

yeah right, that sounds painful.

good on you for supporting it though, it'll be quite the feature.

[stone igloo]

do you always do THP?

[nova fiber]

what does THP stand for?

transparent huge pages?

since those are the 1g pages no i dont do that yet

I'd like to at some point

[stone igloo]

no 2mb is also THP

[nova fiber]

oh, in that case yes i do THP then

but only with 2m pages

[stone igloo]

cringe

:^)

[nova fiber]

partial unmap of 2m pages is borderline manageable to do without creating spaghetti, partial unmap of 1g pages would be turboaids ngl

[stone igloo]

partial 1g unmap requires allocating

[nova fiber]

yeah but it would be require up to 4 extra page tables being allocated

which is just going to be messy to do upfront

[stone igloo]

why 4?

its up to 511 i think

[nova fiber]

if you unmap the last 4k and first 4k of two adjacent huge pages thats only 4 allocations

2 pdpts and 2 pdts

[stone igloo]

huh?

oh

[nova fiber]

sorry pdts and pts

[stone igloo]

uhh

[nova fiber]

one level off my bad

[stone igloo]

just call it by number

lol

[nova fiber]

how would it need 511 though

unless you're doing insane stuff like scatter map and unmap

[stone igloo]

you unmap in the middle

and you need to turn a full 1g page into 512 2m pages

oh wait no thats one alloc

yeah im stupid

lol

[nova fiber]

i have thought about this at great length

anyway 2 phase commit is bad enough with 2 allocations to pass around

2 phase commit with 4 allocations on two page levels would be really finnicky to get right

[nova fiber]

i think im happy with the test coverage for now

i've got nearly 100% branch coverage and i test some of the oom cases

i know branch coverage isnt a great metric for actual test coverage but its decent enough for this i think

[woeful raptor]

Can I ask how you got Google test unit testing working with QEMU? Or do the tests not run in QEMU? Tried googling for tutorials but got nothing.

[nova fiber]

i run the tests in userspace

i have kernel test images as well that i do run in qemu but no integration with gtest yet

[woeful raptor]

Meaning you just test code normally with gtest and not thru QEMU?

[nova fiber]

yes

most of my kernel builds on both hosted and freestanding so i can test it without needing a full kernel

i use libvirt for doing automated testing with qemu but thats for integration testing rather than unit tests

[woeful raptor]

I see I am trying to see if I can get unit tests to work with QEMU because I want to test my PMM but according to osdev wiki it's not really unit testable.

[nova fiber]

i dont see why not, i test mine in userspace just fine

its just an allocator

[woeful raptor]

Don't you need to QEMU to emulate memory for you?

[nova fiber]

you dont need to emulate memory to just test a pmm

even if you're storing memory in the pages themselves just tell your pmm everything is identity mapped and give it a few bits of memory from malloc or mmap

i test my entire paging system in userspace by basically doing that

[nova fiber]

moving onto implementing mapping correctly in oom cases

currently if i run out of memory for the page tables midway through mapping memory the range is left partially mapped

so now im computing the required number of pages early and then allocating them upfront to reserve them

i've implemented a very simple fixed size freelist allocator for page tables that allows deallocating singular elements so i can do a single allocation and then deallocate parts of it as needed later

[nova fiber]

oom really makes everything harder it seems

[brave cape]

why do you need a separate allocator for page tables?

[nova fiber]

it saves alot of headache when pre allocating page tables

[brave cape]

one thing that would be possible is to do strict accounting for some things, e.g. page tables, and overcommit in other places

although this is in essence moving the work of keeping track of the preallocated page tables into the page allocator

[nova fiber]

i dont currently have overcommit since no disk writing, but the seperate allocator is more for the partial deallocation behaviour than anything else

most malloc implementations are quite unhappy with being given pointers partway into an allocation and being told to free that

this means i can allocate like 10 pages in a single malloc and then deallocate them 1 page at a time as they're no longer needed

[brave cape]

what, do you allocate page table memory as higher-order pages?

[nova fiber]

right now i have a fixed size memory pool for the kernel and each process that i allocate page tables in

im aware its not ideal

[brave cape]

yeah, it's a bit of a "wtf?" thing

because normally with buddy-like systems, you can free at any (>=2**12) granularity

[nova fiber]

i'd eventually like to have flexible sizes for the memory pools but im trying to not rush things too much

[calm shell]

Why are you allocating page tables at map time at all

[brave cape]

yeah, not rushing things out is probably a good idea

[nova fiber]

when else would i allocate them?

[brave cape]

in the page fault handler

[nova fiber]

at the point I implement a page fault handler that doesnt just kill the process i will do that

[brave cape]

the concept is quite simple actually. you basically do the following

if userspace pagefault or copying to/from userspace:
        grab mm lock
        vma = find vma
        if !vma or deny write or deny exec:
                ungrab mm lock
                deliver SIGSEGV
        else:
                fault-in the page
                ungrab mm lock
else:
        die

[nova fiber]

i know how to do in theory, i just dont have any disk driver to store a page file so everything is comitted

[brave cape]

just do strict accounting, then :^)

[nova fiber]

isnt strict accounting just not letting programs commit more memory than you can guarantee it

that was my plan anyway

[calm shell]

nothing about demand paging (incl lazy page table allocation) requires a pagefile

[nova fiber]

how would you garantee that memory is going to be available to handle the page faults if you arent already reserving it beforehand though?

[sour geyser]

You don't guarantee it, you play chicken with whoever requested the memory and see if they use it.

[calm shell]

nuh uh

you do that if youre linux but he said he wants strict accounting

so thats off the table

[sour geyser]

Ah

[nova fiber]

i very much want to avoid the oom killer

so either paging or memory compression are what im probably going to do next

well not 'next' exactly, but at some point

oh not sure what i did but i fixed ps2 input on vbox

i really need to replace the default flanterm font with the vic20 font

incredibly hyperv actually caught a bug that no other hypervisor complained about

it actually verified that writes to IA32_FSBASE, IA32_GSBASE, and IA32_KERNEL_GSBASE are canonical addresses

nothing else verified that

[willow snow]

WWWWW

[nova fiber]

hmm on my laptop vbox ps2 keyboard input is glacially slow for some reason

cant reproduce on vmware, or on my desktop either

everything else runs at regular speeds but the keyboard irqs come in like once every 2 seconds

[nova fiber]

shitty strncmp shell i know but the underlying syscalls are the thing im testing here

[olive musk]

bad

use strncmp at least

!!

[nova fiber]

are you blind

[olive musk]

bruh I read strcmp

lmao

[nova fiber]

your brain makes you think

[olive musk]

my brain is unsafe

smh

[nova fiber]

once i get a working sysfs the next thing i'd like to do is split the shell from the tty and have the tty as a device that the shell can just open like a regular stream

sysfs comes first so i can actually look at all the installed hardware to figure out what drivers i need to implement next

right now i've been doing it by putting halts in various points of my boot process to be able to read the walls of output

[nova fiber]

working sysfs for smbios tables

the smbios folder itself is both a folder, smbios node, and file

so you can cd into it to see its children, use show on it to see the toplevel properties, or cat from it to see the smbios table binary data

[nova fiber]

also put acpi in there

[urban schooner]

what does the show command do

[nova fiber]

queries the identify interface of the current node and prints its info fields

[urban schooner]

is it some sort of a syscall

what does show do in a normal directory?

[nova fiber]

in a normal directory it shows the name of the directory, name of the fs mount, and vendor of the fs mount

everything is a device, so you can use show on anything

[urban schooner]

interesting

so is it a syscall?

[nova fiber]

it uses a syscall, but its not a syscall specifically for that no

[urban schooner]

whats it called?

[nova fiber]

i have eOsCallDeviceOpen and eOsCallDeviceInvoke

[urban schooner]

like ioctl?

[nova fiber]

eh sorta, its a bit more strongly typed since you need to query a specific interface from a vnode before invoking methods on it

[urban schooner]

query an interface?

[nova fiber]

rather than just opening /dev/kvm and doing ioctls on it you'd need to query the virtualization interface from /dev/kvm and then invoke methods on that

i dont have files or folders, everything is a vnode with interfaces that you query

[urban schooner]

hm

interesting

so to open a file for reading you need two syscalls basically?

[nova fiber]

you'd query the file interface from a vnode and you can add parameters to use when opening the interface

opening a node and querying an interface are one syscall

[stone igloo]

thats cool

can you open another interface on an already open node?

[urban schooner]

so u open and query at the same time?

[nova fiber]

i'd like to provide the option to do it in two calls so you can query multiple interfaces without doing the path walking twice

[nova fiber]

right now yes

[urban schooner]

ah makes sense

[nova fiber]

actually that might be a better idea, yeah

[stone igloo]

lol

[nova fiber]

every vnode implements the 'identify' interface, so you can use that as a sort of generic handle

[stone igloo]

are the interfaces named?

or just ids

[nova fiber]

that interface just has the device info fields call, and a call to get the other supported interfaces

the interfaces are just guids

i'll add something to turn the interface guids into nice display names at some point

[urban schooner]

guids make sense

[stone igloo]

cool

uefi style :^)

[urban schooner]

yeah

[nova fiber]

this does let me do some neat stuff for system management, from the command line you can cat a file like one of the acpi tables and get human readable output. but as a program you can open a specific table interface and query the fields directly without needing to parse text

[stone igloo]

thats neat

does your cat do something like powershell where you have typed objects?

[nova fiber]

i like everything being a file from the command line from a sysadmin perspective, but parsing text in programs is painful so providing both with this is pretty trivial

[stone igloo]

or is it a straight up textified format

[nova fiber]

currently cat is just straight text

im not sure if i would want to provide a typed data view from the device directly or if it would be better to have userspace command line programs that can interpret the object data and produce something similar

[stone igloo]

imo you should have a typed view

[nova fiber]

mmm getting all the required data represented in a flat object seems like it would be a little painful

i'll give it some thought

[stone igloo]

you don't need a flat object tho?

just allow properties to have subproperties

[nova fiber]

i meant more the byte representation than conceptually

im trying to avoid pointers inside the request packet as much as possible

[nova fiber]

once again made syscalls pre-emptible

had to allocate a syscall stack per thread rather than per cpu core

guess who spent 2 hours debugging a copy paste error

[nova fiber]

more memory corruption yipee

[calm shell]

make literally everything preemptible except the scheduler and interrupt handlers (and only bc they structurally cannot be)

[nova fiber]

thats what im doing now

[calm shell]

and even those should be interruptible

!

[nova fiber]

guess who forgot to set the ist stack for irqs

the tty and shell are now seperate processes

i think i may have to start stripping debug symbols at some point

or not build with -O0

[nova fiber]

quick stop to work on my repo tool a little, i need to add support for multiple configure steps so i can run autoreconf when needed

[nova fiber]

also while im at it i think i'll change the internal representation to make everything a target

[nova fiber]

done the first part

[nova fiber]

tried switching to using rsp for interrupt handling but it seems that my scheduler is not happy when i do that

i think whats happening is during an interrupt the scheduler reschedules the interrupt handling thread to another core and then everything explodes

[nova fiber]

i have a few other hard to track down bugs so i think im going to do some long running tests with libvirt and add some instrumentation using the second serial port

[nova fiber]

testing™️

didnt know the alt tab menu had a scroll bar

[nova fiber]

ok so my rcu implementation was unsound

i now deadlock after some time, although i think its actually the scheduler forgetting about the thread again

but no crash so its an improvement

the debug messages over the second serial line are useful for high volume stuff like scheduling events

hmm no its not forgetting the task

so i guess this is a deadlock in userspace

the horror

maybe in order to write some unit tests for my core process/thread/mutex stuff

i will also distract myself by porting ncurses so i can port zsh

[nova fiber]

holy shit ncurses builds for bezos

i dont imagine it works at all because everything it calls is just stubbed

anyway i think thats enough for today

[verbal vapor]

you find a window secret

[nova fiber]

spent 2 hours sorting a linked list award

its harder when the node data is the node itself

its a freelist and i decided to be clever and save space by putting the freelist node at the start of the block it marks

anyway back to porting zsh

[calm shell]

whyd you need to do this

[nova fiber]

i defragment the list by merging adjacent freelist entries, and its easier to do that when the freelist is sorted

at least now i've solved my problem of producing oom errors despite there actually being enough memory to allocate page tables

[calm shell]

is this for your heap or pmm or what

[nova fiber]

this is for the heap specifically for allocating page tables

because i want that partial free behaviour

[calm shell]

partial free behavior?

[nova fiber]

like i can allocate 20 page tables all at once, and then free the entries one at a time

[nova fiber]

man config.guess and config.sub are so weird

i can see why llvm basically forces you to use the 3 part of 5 part triple and doesnt accept anything else

i really dont like that both autoconf and automake contain these scripts are they're subtly different

[nova fiber]

ree zsh doesnt include the correct headers

uses strcasecmp but doesnt include strings.h

[nova fiber]

zsh has built and linked

god only knows if it works

time to find out

[nova fiber]

ah since i improved my oom handling in my page table stuff im now actually catching errors i didnt catch before

whoops

[nova fiber]

shockingly my elf loader actually survives loading zsh

zsh itself immediately aborts

but like, no triple fault

so i'll take it

[nova fiber]

well it runs, dies, and then control returns to the shell

im quite pleased with that

[urban schooner]

Do you have multiple shells?

[nova fiber]

i have my crappy shell thats just a pile of strncmp for testing

i'd like to get an actual shell though, hence porting zsh

[urban schooner]

Lol

Makes sense

Why zsh over bash

[nova fiber]

bash source is turboaids

zsh is much less gnuoided

[urban schooner]

Hm

[nova fiber]

i tried porting bash first but decided it wasnt worth it at the point bash decided to declare its own prototypes for a bunch of posix functions that arent compatible with the posix spec

[urban schooner]

Lol

Linux is the only platform that exists anyway

[nova fiber]

my favourite part of bash is that in the configure script it detects tons of stuff and tried to reimplement it if its not found, but most of the reimplementations are broken and dont compile even on linux

[urban schooner]

Lol

[stone igloo]

zsh is the better shell anyway

tbh zsh is probably a better choice because they have cool extension stuff

[forest lily]

don't you mean ksh?

honestly tho ksh would be my goto shell to port

[nova fiber]

huh using -ffreestanding implies -mno-sse and -mno-mmx on clang

thats certainly a choice

[tropic berry]

wtf

since when?

[nova fiber]

guess i need to enable sse and implement xsave

idk, im using llvm 21 but havent ever tested this specific edge case before lol

suddenly got UD exceptions after removing the freestanding flag from my init programs build

sure enough movups

removing -mno-sse and -mno-mmx didnt do this, i removed those last week and it was fine lol

[nova fiber]

intel engineer sneezed again

[nova fiber]

xsave is proving to be more complicated than I initially thought

[nova fiber]

is xsave not enumerated from cpuid.1

even with kvm enabled bit 26 of ecx is zero

[nova fiber]

(this was fixed by qwinci, i needed to specify -cpu host or +xsave)

but its good that its not always enabled so i can test fallback code

[nova fiber]

another banger diagram from the sdm

[nova fiber]

xsave and fxsave work now so i can stop passing -ffreestanding to all my userspace programs

now back to zsh

[nova fiber]

im not really sure how i want to go about doing stdin, stdout, and stderr

my first thought is to maybe have the process being created be responsible for creating the files inside their own procfs folder and then the creating process waits for all those files to be created

although that doesnt seem great

[calm shell]

like

fds 0, 1, and 2?

[nova fiber]

yeah like how they're created when a new process is spawned

i dont really want the kernel to be responsible for creating them

[calm shell]

theyre inherited from the parent

by default

all fds are actually

(if not otherwise specified with something like O_CLOEXEC (iirc is what its called))

well actually CLOEXEC closes it upon exec() hence the name

not upon fork()

[nova fiber]

ok maybe spawned was the wrong word

maybe the word i was looking for is exec'd or CreateProcess'd

inheriting the parent processes fds just seems strange to me

[calm shell]

well its how stdio works

on windows the handles arent inherited by default iirc, but the stdio handles are explicitly copied into the child process before its set running

with NtDuplicateHandle or something like that

[nova fiber]

maybe i'll go with that then

im not going for perfect unix compatibility

[calm shell]

are you nih'ing your userspace then

[nova fiber]

partially

i've got a posix compatibility layer but i'm trying to keep it all in userspace

[nova fiber]

added logging to all the posix stubs so i have a place to start now

[nova fiber]

hehe

[nova fiber]

nice pseudocode intel

[nova fiber]

did some cleaning up of my vfs, quite a few bits of dead code lying around and the interfaces for some stuff were pretty nasty

stuff is marginally less nasty now, although it'll take a few more go arounds before im happy

once again my high test coverage pays off

[nova fiber]

despite having working tls in elf i maintain that the elf abi for thread local storage is totally ass backwards retarded

negative offsets, scuffed alignment rules, self referential pointers, its just got every single thing wrong all at once

im a little stumped on folder iteration

my old system was pretty shit, and now that i've killed off the old vfsnode class i cant just grab an iterator and copy stuff out

specifically around procfs this becomes nasty

i want to provide stuff lazily when iterating procfs and ideally dont even want an internal directory structure, instead turning any vfs operations into queries against my internal data

on the bright side i've got an allocator and basic mmap in libc now so i get alot further into zsh before aborting

i do enjoy that c++ lets me abuse so many features

i usualy dont like multiple inheritance but this specific case has been really useful

if i need default behaviour for an in memory folder heirarchy or providing an identify interface i can just inherit from the mixin and use some templated interface handles

this is perhaps my most cursed mixin

a static variables storage address is something you can template on

[nova fiber]

after thinking further about how im going to do stdio streams this might be a good chance to use my typed command line args stuff

rather than just taking an array of strings as command line args programs can also take structured binary data, so i can have the calling process create stdin, stdout, and stderr then pass those directly to the posix runtime inside the program

im not quite sure how i'd do validation for the parameters, maybe something like msgpack or protobufs

[nova fiber]

im starting to think i should enable exceptions

just maybe

[olive musk]

I have a try macro for doing that

it returns the error code (like actually returns from the function) if it errors or just gives you the value

Ah well I guess since you use out parameters that won't work

I have a result type

[nova fiber]

yeah i have a similar macro, its more that having to spam it everywhere is annoying

and with the amount of checks i have throwing exceptions would probably be faster and produce a smaller binary

anyway folder iteration works again

[nova fiber]

hmm yes gcc very atomic

oh thats strange

gcc doesnt make the shared pointer refcount a std::atomic

the ref count is a plain int and they add memory barriers instead

does gcc make some garantees about increment and decrement always being atomic or something

oh thats also weird, when libc++ is compiled without thread support it doesnt use atomic ref counting for shared pointers

[nova fiber]

guess who overwrote their system cc with a cross compiler

luckily i only use this wsl distro for developing bezos

but still, oops

finally getting around to adding a bezos toolchain to clang

its a real pain in the ass generating the patch files ngl

[nova fiber]

actually everything about llvm is a pain in the ass

[nova fiber]

also now splitting building clang and building libcxx into seperate packages

now that i cant just piggyback off system libc++ it complains about missing stuff

[brave cape]

or just

OsStatus status;
// ...

status = context->readObject (userCreateInfo, &createInfo);
if (status)
        return CallError (status);

status = SelectOwningProcess (context, createInfo.Process, &process);
if (status)
        return CallError (status);

or, alternatively:

status = context->readObject (...);
if (!status)
        status = SelectOwningProcess (...);
if (!status)
        status = UserReadPath (...);
// ...
if (status)
        return CallError (status);

[nova fiber]

its not so much the way im doing the handling, its the code that it'll generate

exceptions generate fewer instructions on the hot path since all the handling is in cold blocks

[brave cape]

but imo that's a minor thing

[nova fiber]

over an entire project there can be some real space savings

https://youtu.be/bY2FlayomlE?si=SwTwl9L4VW9z-wrD im mostly referencing this

[brave cape]

btw this is one of the most common misconceptions about exceptions. "it makes code slower"-NO IT DOESNT you fookin...

it makes the error path slower

exceptions should be used when you don't care about performance in the error path

[nova fiber]

cross compiling compiler-rt requires a populated sysroot
building libc to create sysroot requires cross compiled compiler-rt

slightly annoying

[verbal vapor]

just have a make target to put the header wihout building anything

[nova fiber]

i also dont have a sysroot so writing the clang driver toolchain is turboaids

and clang just refuses to change where it searchs for compiler-rt

god forbid i put it somewhere other than directly next to the clang compiler

[nova fiber]

fuck you fuck you fuck you fuck you

ok finally got it working

i now cross compile compiler-rt

while not having a path or sysroot is probably going to cause me alot of pain i will be sticking with it

osdev is if anything an exercise in petty defiance

and i dont like /lib, /include, $PATH, or env

so i will not be having any of that

oh man all my clang tooling has shat the bed

im still using the global install of clangd so it doesnt know about my toolchain

meh whats another 2000 build targets between friends

ough this has broken the kernel build

i either switch to use libc++ rather than libstdc++ or convince clang that im actually targetting a very fucked linux while building my kernel

oh god clang has started using both the system libc and mine while compiling the kernel

incredible

wtf even with nostdinc its still finding the build machine headers

days since last clang crash: 0

[nova fiber]

it is done

god that was unpleasent

and now zsh doesnt load again

my elf loader is so fragile

wow i got fuck all done today except raise my blood pressure

honestly tempted to just revert all my changes

all that just to not build protoc

[verbal vapor]

good luck porting anything then

[nova fiber]

i dont see why it would make anything significantly harder aside from this clang toolchain

im just going to need to specify a few more include paths

either that or create a fakeroot for each program and symlink in the required dependencies

i really dont like the global nature of lib and include because you can update some shared library and not realize its broken something for ages

having either a fakeroot or needing to specify exact paths makes versioning more explicit

[nova fiber]

in other news i've nearly managed to kill off my old fs syscalls

i now have a way smaller set of calls

just vnode open/close/query/stat and device open/close/read/write/invoke/stat

im semi-formalling enfocing that all data sent to invoke must be a single flat object so that in the invoke syscall i can just check the entire thing is mapped and then safely use it in the rest of the kernel

i still have a few other syscalls for stuff like processes, threads, and whatnot

for doing subprocesses i think i actually have a good method now, since all nodes are global they can be shared pretty easily and im just going to enforce that anything that you will share with a subprocess is put onto the processes ACL when the process is created or added to your own ACL and granted access to your child processes

also my main.cpp file is now so large clangd stops working so my hand is forced to split it up a bit

[umbral grotto]

i seriously thought that was a typo of repconf for a sec

the interrupt handler saves most of the state already

[nova fiber]

holy necropost lmao

i solved that like a month ago lol

i mean im still using ists for everything because i stack overflow when using rsp0 but whatever

[umbral grotto]

wow my bad

[nova fiber]

nah i've done worse lol, sometimes discord wont show me the timestamps on messages lol

i've replyed to stuff from like 2 years ago before

[umbral grotto]

i just don't notice

[nova fiber]

split apart elf loading enough to let me pass extra arguments to process creation

now i can start passing creation parameters from userspace

im not particularly happy with the kernel being the one doing the elf loading but for now i think i'll stick with it

ideally i'd like userland to be in charge of most of the program loading stuff

[nova fiber]

clang is actually the worst

every day this toolchain does something strange and fucky

finally got around to adding git overrides for repobld

sick of generating patch files manually, now it can optionally clone a git repo instead of downloading an archive

so i can get git diff to generate the entire patch

also how does llvm already have x86 APX support

how are they testing this

[calm shell]

did intel implement it

if so then probably they have a simulator or extremely early test hardware or both

[nova fiber]

ah it would appear so

im used to corporate contributions usually coming from accounts called like john-ms or bob-amd

this is just phoebewang with no intel suffix

[stone igloo]

i think this is pretty common

[nova fiber]

i guess it depends on the company

[stone igloo]

pohebe wang is also one of the two x86 maintainers

so

[stone igloo]

which project has things like that actually?

[nova fiber]

most of the microsoft repos, i think all nvidia ones, as well as amazon, oracle, and ibm

[stone igloo]

interesting

[nova fiber]

everytime i've contributed to oss at work i've been given an elliothb-work github account

[stone igloo]

i see

[nova fiber]

i mean in the sense that it was part of my job to make the code change for work, not just that i happened to be working there and corporate policy says so

[stone igloo]

yeah i see

[nova fiber]

Fuck this i am binning the last 3 days of work

I will cherry pick the few good bits and do something else for a while instead

Clangd has been fucked for 3 days and I can't figure out why, not having any tooling is a real pain and the code I have written has a generally unpleasant vibe

There's a few decent bits I will be taking but all the llvm bullshit has got to go

[nova fiber]

finally root caused clangd exploding

my kernel builds with clang 21, my system clangd is clangd 20

damnit

[nova fiber]

finally fixed everything

christ

longest 4 days since i got oracle rac running in docker

i now build 2 copies of libcxx and compiler-rt

one for build and one for host

the build machine one is all built at once, and then the host one is built in 3 seperate parts since parts of it depend on libc

also stubbed out a while pile of random shit to get it building

wchar.h exporting struct tm and FILE is retarded

[nova fiber]

another day another elf loading bug

now the segments are overaligned

so i need to round down instead of up?

or do i need to round to the closest multiple

[sour geyser]

Align down the base, and align up the top

[nova fiber]

I think i've finally worked out the alignment bugs

now i have stack alignment issues instead

checking the return value of vsnprintf fixed it

slightly concerning

clang generates a movaps followed by a movups and both use the stack as memory operands

the stack isnt aligned to 16 bytes and then it explodes

[nova fiber]

got exceptions and unwinding working in userland

only using the libunwind baremetal mode since no dlopen yet

also stubbed out yet more functions in posix so i can now use libc++

methinks my elf loading is still a touch fucked

hang on thats my kernel symbol table

oh dear

[nova fiber]

hmm clang actually isnt aligning the stack properly

[nova fiber]

yeah stepping through with gdb the stack pointer actually isnt aligned to 16 bytes on function call boundaries

wtf

i pass an aligned stack pointer to the program main so im really not sure what gives

do i actually need to pass the stack 8 bytes misaligned, the first instruction in the entrypoint is a pushq for the frame pointer

ah of course

actually that didnt fix it

damnite

nevermind it did my build was stale

[stone igloo]

wtf is gnu::force_align_arg_pointer

making an unaligned pointer is ID

and dereferencing a pointer which was ever unaligned is UB i think

ah force_align_arg_pointer is about stack?

cursed

[nova fiber]

i think im going to spend a bit of time working on finally testing interrupts and the scheduler

and doing some multithreaded testing as well since im pretty sure theres some race conditions i havent covered

and also implementing a mutex that isnt while (!try_lock()) yield();

[nova fiber]

got some basic interrupt testing

now to finally emulate mmio with that horrible ptrace setup

also using this as an excuse to implement some wrappers around the debug registers

[nova fiber]

it is not going as well as i hoped

[nova fiber]

success

i "only" needed to install a segv handler to detect mmio access, then disassemble the instruction with capstone, fork the test into a child process, install a breakpoint on the next instruction, then have the child ptrace the parent and forward the signal

but now i can test the apic, hpet, pcie msis and msi-x, and the scheduler all from the comfort of userspace with nice debugging tools and unit tests

a day well spent i think

certainly better than last week

unfortunately valgrind doesnt work on it but thats a worthwhile cost

oh and code coverage shits the bed

[nova fiber]

fixed code coverage

i think this might be first actually novel thing i've managed to do with osdev so far

afaict no other kernel has managed to test mmio stuff like this

and since i have capstone i no longer need to make all my port io stuff cluttered with #if __STDC_HOSTED__ i can just wait for SIGILL and read the operand values

also works for crN, drN, and anything else i want

hell i could even test my syscalls directly now, i can emulate sysretq and iretq

[nova fiber]

tada

dont mind the weird pointer arithmetic

[sour geyser]

this is cool, I'm feeling inspired

[olive musk]

How do you ensure proper emulation of devices?

Like if you want to test the ioapic or whatever

How do you know your ioapic implementation is correct

[nova fiber]

good question

right now i just read the spec carefully and hope its right

in the future maybe i'll link in qemu or virtualbox devices instead

if im feeling really extra i could also emulate paging using the segfault handler and memfd_create

[olive musk]

cuz rn I feel like you're not really testing anything?

you're writing values

but how do you know they're right

[nova fiber]

well right now i just compare against what happens on real hardware

once i get the interrupt stuff setup i can then do more proper functionality tests

to begin with im just using this for regression testing

this is mostly just setup that i needed as proof it would work at all

there was also a bit more to that apic test than in the screenshot, i do have verification that the icr writes happen in the right order, alignment checks, and i abort on writes to readonly registers and reads to writeonly registers

once i get virtual memory emulation i'll be able to emulate my entire boot process which will be much more useful

[nova fiber]

this is a bit of a problem in testing lol, there always needs to be some bit of code thats "dont worry about that we're sure its right" because of the chicken and egg problem

[olive musk]

yea but you get what I mean

[nova fiber]

yeah i do

[olive musk]

happens to me too, sometimes I test stuff and I realize I'm just testing for the sake of testing

[nova fiber]

im getting this stuff setup so i can test my scheduler mostly

doing soak tests with 200 copies of qemu isnt really feasible

my scheduler interacts with the apic hence i need to emulate mmio and msrs to emulate that

i have a very strong policy of not changing implementations for the sake of testing

beyond maybe putting some functions in a detail namespace that would otherwise be static inside the source file

[nova fiber]

i am also testing a little for the sake of it because i enjoy doing kinda obscure stuff like this

[brave cape]

what you haven't realised is that you are not writing a kernel, you are writing a hypervisor

[nova fiber]

i thought this was closer to paravirtualization since its all running in a single userspace process

[brave cape]

if this goes on for long enough, you will eventually decide to put the kernel in kvm to intercept mmio

[nova fiber]

shockingly my os doesnt die on oom

i leak a shitload of memory but somehow i fail gracefully

i think im just going to focus on stability for a bit and try to track down some of the more egregious memory leaks

[nova fiber]

implementing page table compacting correctly first time

shocking

not my proudest moment but oh well

[nova fiber]

also implemented partial remapping of large pages

previously i'd just unmap the area then map it again internally, now i actually detect its a 2m page and shred it into 4k mappings during the map

still no support for 1g pages yet

[nova fiber]

how do i pin a thread for myself, everytime i dont talk in here for 12 hours discord hides the thread for me and i have to do digging through the channel to look for it

really annoying

[nova fiber]

oh yeah and finally got around to training the apic timer so i can do sleeps

i'll also do the same for invariant tsc since amd isnt kind enough to provide cpuid 0x14 for the clock frequencies

oops my math was wrong

2401260000hz is damn near identical my cpus base clock so i think thats probably the right tsc frequency

i should probably add some pretty printing to the frequencies like i did for storage units

[nova fiber]

good enough

[sour geyser]

I'm sure this was a typo, but please tell me you haven't been checking leaf 0x14

[nova fiber]

0x15 my bad

and 0x16

[nova fiber]

bruh i spent a whole day getting the hpet stuff working and now i go to check linux source and the hpet is basically immediately disabled and never used

wtf

actually its used as the fallback of last resort if nothing else is present aside from the 8254

still though

damnit

also for some strange reason __rdtsc() is a builtin on clang but not __rdtscp()

not sure what the reasoning behind that was

oh and i found a bug in the c++ standard, std::span<volatile hpet::Comparator> doesnt compile because common_reference_t doesnt work with volatile qualified types

it does work if you declare volatile and const volatile move and copy constructors

[nova fiber]

another thing off the todo list

it uses the invariant tsc if its available, and will fall back to the PIT if i cant find an hpet

[brave cape]

surely you mean __builtin_rdtsc and __builtin_rdtscp?

[nova fiber]

clang has __builtin_ia32_rdtscp, but __rdtsc is a builtin

[brave cape]

yeah, ia32_*

but yeah

[brave cape]

__rdtsc is probably there for compat with MS bs

[nova fiber]

yeah maybe

microsoft does have quite a few useful intrinsics that the mmintrin headers dont define actually

its quite annoying that there isnt an intrin header for the supervisor mode instructions aside from the microsoft compat one

[sour geyser]

Lol I think of this comment everytime I do timer-related things

[tropic berry]

the PIT was honestly the last remaining sane timer for doing calibration and now that can't be relied on anymore

i mean i guess one could use the ACPI timer...

at some point you have to think hardware people are just sadists

that do it on purpose

[nova fiber]

i use the hpet for training the apic and tsc and as a fallback incase invariant tsc isnt supported

huh the acpi timer is yet another thing

you'd think every pc is 20% quartz with how many timers they have

oh and just to be extra i use rfc9562 as my reference epoch and time step

anyway today i think i will do some more code cleanup and start moving over to my new task system code

my current task management does not inspire joy

oh yeah and finally remove my old file syscalls

and if i have time figure out why clangd just dies when i open any of my userspace programs source code

jump to definition just gives up despite clangd not crashing

[nova fiber]

i think i finally figured out why the debug log messages im getting from userspace sometimes contain kernel debug symbols

i dont zero pages before returning them, and rpmalloc expects zerod pages

so im giving it physical memory that used to contain the kernel elf file without clearing it

oopsy

[nova fiber]

removed the old file syscalls, everything goes through the node and device calls now

still not sure why clangd dies

also thinking of redoing my toolchain stuff, maybe using symlinks and fakeroots rather than specifying paths individually

i still dont want a global sysroot because thats a great way to accidentally depend on the wrong version of a dependency and then break stuff after an update

which i actually ran into when i uninstalled my system copy of capstone so i could link against the version i built only for qemu to stop working because it depended on the system copy

very annoyed that afaict no distro ships debug symbols outside of debuginfod

apt install library-dbg when

I think as part of my task system rewrite im going to move the tls data handling into userspace

handling it in the kernel is just annoying and wont work once i get past static linked executables

[nova fiber]

i think i've finally made my rcu based atomic shared pointer implementation sound

i think i will hold off on implementing hazard pointers for kernel objects but i'd like to do that at some point

hazard pointers are very tricky to use right

[nova fiber]

factorio is actually heroin good god

i like that one of the suggested methods of ipc for win32 on msdn is using the clipboard

like sure technically that is ipc, but i feel like its somewhat suboptimal

[nova fiber]

ooooh fb folly has a hazard ptr implementation

maybe i will use that, it looks reasonable

this looks pretty similar to rcu but with more fine grained read announcements

[stone igloo]

i wonder how clipboard ipc actually works

[nova fiber]

iirc its managed by dwm and its just a big unstructured block of bytes with some metadata attatched for what a program says was pasted in there

actually nvm its a com server, but still just a big blob of bytes

[nova fiber]

https://computernewb.com/~lily/files/Documents/NTDesignWorkbook/ found some decent reading on the windows object model finally

more comprehensible than greping through nt5src for Ob*

[calm shell]

I coulda shown u that

[nova fiber]

darn

dont suppose you have anything on the nt scheduler

[calm shell]

I even have the DEC Mica design notes that those are poorly summarized from

[calm shell]

Read Windows Internals

[nova fiber]

i will give that a read tomorrow

i've mostly been going off "An introduction to operating systems" first edition by Harvey Deitel

[olive musk]

Inside windows NT is really good too

It's less recent however

[nova fiber]

started writing tests for the process management stuff

think im going to have to finally kill off SystemMemory because its just so unweildy

[iron swallow]

doesnt cover big.little hybrid scheduling :((

windows internals is more about windows 7/8 scheduler than windows 10 or 11

[nova fiber]

about halfway through killing off this damn class

i think it was basically the first thing i wrote that wasnt in kmain so alot of stuff uses it

[nova fiber]

i think i've killed it off enough

theres a few places where it still makes sense

[nova fiber]

first test with the new process system and its alot nicer to add features to

time to write yet another allocator, this time for managing physical memory

because my old one doesnt feel very good

i also realized that d3d12ma and vkma basically implement exactly what i want so i can just port d3d12mas CreateVirtualBlock implementation and i have an allocator that can manage physical memory very well

[nova fiber]

i've spent like 2 weeks making this damn process migration happen

it'll happen eventually i swear

I've mostly yoinked the NT design since that matches better, but im making process creation almost entirely up to the userspace

creating a process object does nothing beyond that, the process doing the launching has to create the appropriate virtual address layout, and setup the main thread

probably isnt going to be as quick as the unix model since its going to involve a fair few syscalls to do all the setup but means i dont have to do as much program loading in the kernel

[nova fiber]

got threads and processes done, will do virtual memory mapping and devices tomorrow

hopefully scheduling soon-ish as well

im not entirely happy with how process destruction can currently leave stuff in an inconsistent state if it fails partially

i guess i should get around to transaction support sooner rather than later so i can implement everything in terms of transactions rather than the manual rollback work i have to do currently

[nova fiber]

transactions are a touch complicated it turns out

for them to be useful i basically need to support nested transactions at a minimum

and i need isolation modes for both reads and writes

[nova fiber]

also i think im going to adopt an error reporting api similar to ibm db2, it allows much better reporting than just plain error codes which is nice

in some code paths theres probably 20 different places that can return OsStatusOutOfMemory and it would be nice to know if i run out of physical memory, virtual address space, space to allocate page tables, or some other form of memory

and i imagine with transactions im going to need much more detailed error reporting than just "OsStatusTxAborted" when a transaction cant be comitted

[nova fiber]

burnout

anyway i've finally started moving to the new process and object model

i'm going to move elf loading into a seperate library that will be used in both userspace and kernelspace

in kernelspace it will only be used to load the init program, after that the init program needs to use rtld

[nova fiber]

its really going well

[nova fiber]

i think i finally need to bite the bullet and implement fakeroots for builds

linux allows for multiple readonly overlay filesystems so it shouldnt be too hard, i just need to clone into a new user namespace to use mount without root

[nova fiber]

yknow despite fork and exec basically being the recommended way of doing multithreading on unix systems basically every single library keels over when you even dare try

glibc malloc shits the bed if you dare clone with CLONE_VM

[nova fiber]

ah using clone and fakeroot breaks tooling really badly

onto plan b which is just making a fuckton of symlinks

[nova fiber]

ugh clang keeps exploding

nope not dealing with this

i'll stick with the kinda shit sysroot i have now

i'll deal with this some other time

[nova fiber]

yipee i think i've hit a meson bug

either that or two things are interacting that i dont know about

it would appear sys_root and pkg_config_path somehow interact with each other

it would appear this is mostly not documented and that theres an open bug to document it lol

ough thats a strange behaviour

pkg_config_path is used to search for pkgconfig files, which is fair enough

but if you specify sys_root that isnt prepended to the pkg config path when searching

but it is prepended to the pkg config output if it uses ${pcfiledir}

[nova fiber]

ough chicken egg problem strikes again with the rtld

actually i can just split posix into multiple libraries

[nova fiber]

so the init program contains a statically linked copy of the rtld, which i then use to launch a shell process with everything dynamically linked

and then from that point on everything is dynamically linked using the rtld shared object

[brave cape]

what exactly is your idea of "transactions", and how are they used in your project?

[nova fiber]

I think mostly the same as what the term means in a database context, some collection of actions that are all either performed or not performed. and from outside the transaction its impossible to observe an inconsistent state

[brave cape]

how is it used in kernel interfaces, how does it e.g. remove the need for manually rolling things back, as you mention here?

[nova fiber]

most kernel interfaces can accept a transaction handle, if there is one then the action is put into the transaction. then the transaction can either be committed or rolled back

it removes the need for manual rollback because it presents a generic api for it, which each thing that can appear in a transaction having a way of rolling itself back in isolation from everything else

[brave cape]

ahh, ok.

do you have some example code that shows how this simplifies things?

[nova fiber]

in terms of my own syscalls not currently, in terms of databases i have plenty

it will (i hope) let me do stuff like ```cpp
tx = new_transaction();
write_update_to_important_file1(tx);
write_update_to_important_file2(tx);
if (some_error_condition) {
rollback_transaction(tx);
} else {
overwrite_important_boot_data(tx);
commit_transaction(tx);
}

[brave cape]

something like this example is probably impossible without special filesystem support.

maybe a normal journaling filesystem is enough

[nova fiber]

enough filesystems support transactions in some form that i should be alright

im only really planning to use zfs since that has atomic updates

ntfs has transactions as well

in more immediate terms it lets me do stuff like create a process object, map in all its virtual memory, setup the main thread, and then launch the newly created process all at once

and if a quota is hit or theres not enough memory its all rolled back without any extra hassle

[sour geyser]

The idea of transactions is nice, it's something I wanted to explore at some point - so I'm interested to see how you go about it.

[nova fiber]

transactions are great in everything that supports them honestly

it makes so many things way easier to implement

[stone igloo]

i think you can lower this to posix fs semantics in the worst case scenario of forcing slow compat ops

assuming you aren't doing cross-volume operations

if you are doing cross volume disk io, well, i wish you good luck

[nova fiber]

im not quite that brave

[nova fiber]

finally swapped the filesystem over to rcu

rcu_domain and hazard_pointer are both part of c++26 as well now

they were quite brave and mandate an intrusive base class for both rcu and hazptr which is nice to see

[nova fiber]

actually this isnt very hard

zfs does it with a commit log

you just wrap your transactions in a second transaction

you do all the transactions on each disk, write the transaction to the log, then do switchover once the transaction is recorded

if you lose power before comitting to the log the transaction will be cleaned up since the blocks are orphaned

if you lose power after committing to the log then when starting up again you apply the transaction since the log has the required info

[nova fiber]

removing the global lock from my filesystem made it slower

[calm shell]

extra path length from new locks makes it worse if contention wasnt bad to begin with yea

[nova fiber]

it looks like thread sanitizer causes the slowdown

[calm shell]

or that

[nova fiber]

the extra checking on spinlocks makes contention very very expensive

yeah i think you're right, tsan just made it really pronounced

without tsan its within margin of error

[nova fiber]

wait lmao 95% of my tests runtime are spent in std::random

generating random file names and selecting them at random is what is taking so long

enabling tsan also enables ubsan

the random engine does a ton of math and thats then all overflow checked

https://tenor.com/view/rrane-battal-rrane-battal-kızgın-battal-kızgın-gif-25355985

[brave cape]

bro just do a hand-rolled LFSR or something.

LFSRs are like ~10 CPU cycles, possibly less

[nova fiber]

yeah but std::mt19937 and std::uniform_int_distribution are 0 lines of code to write lol

[brave cape]

static uint64_t state;  // initialized with actual randomness
bool
getrandombit (void)
{
        uint64_t bit = state & 1;
        state >>= 1;
        if (bit)
                state ^= magic constant that can be looked up online;
        return bit;
}

[brave cape]

ok probably ~10*N where N is the number of random bits you want

(not taking into account possibly generating multiple bits at once)

still much faster than mt19937

[nova fiber]

true but also just not doing dumb shit fixed the issue lol

[nova fiber]

time for transactions

i think the best way of ensuring proper isolation is diffs and using views for readers

[vestal root]

jeff bezos 💀

[nova fiber]

https://tenor.com/view/jeff-bezos-bezos-my-beloved-heart-locket-meme-gif-25138490

[vestal root]

💀

[nova fiber]

i think i've nearly got the process stuff to a point im happy with

im also taking a quick break to implement mutliarch builds for pkgtool

[nova fiber]

xml is so good when you dont have someone in your ear telling you its shit

[nova fiber]

i think im also going to bite the bullet and write a daemon that runs just to do overlayfs stuff

[vestal root]

[nova fiber]

transactions arent currently fully implemented but i think its finally time to switch over

[nova fiber]

ah fuck i forgot to implement the filesystem calls

oops

[nova fiber]

so many syscalls

[nova fiber]

man why does no arch have write only pages

they're useful

[nova fiber]

you'd think it wouldnt be hard to implement in hardware either

[small stag]

why would write only pages ever be useful?

[verbal vapor]

well if you can only write , you can'r read so you could just ignore in your handler

[verbal vapor]

i guess for something like frame buffer or when program use mmap to map a file as writeonly

[small stag]

But they need to be read tho

Writeonly makes no sense

[verbal vapor]

no if they write to a framebuffer

or if they have shared memory system

the memory can be map as readonly on some process and writeonly on ano

[nova fiber]

Yeah framebuffers or data upload to a device

[verbal vapor]

you could intercept the write (and mark the page as not present) and then write yourself

[nova fiber]

It's more a performance thing to prevent accidentally reading from the area so doing it all in software defeats the point, although it would be useful for debugging

At least in C++ is have a WriteOnly<T> that makes it harder to shoot myself in the foot

[nova fiber]

finally starting the move to the new system api

only took like a month

fuck me

still havent got the rtld working either lol

[nova fiber]

im hoping the rtld wont be too hard since i can map process address spaces into each other

seems much easier to have the kernel load and relocate the rtld then have the rtld map itself already relocated into each process when launching it

[nova fiber]

oh i finally have proper sleep and mutexes rather than busy looping

std::priority_queue my beloved

[nova fiber]

slight aside, i think SBE might be a good option for my ipc message format

if not SBE then i think i'll go with capnproto if i can find a C runtime for it

[nova fiber]

sun XDR may also do what i want actually

[zealous mirage]

it's well regarded and remains the root of NFS to this day

[nova fiber]

my father suggested it to me, he seemed quite positive about it

xdrpp looks pretty mature and aside from requiring some stl types i dont have seems easy to build freestanding

[nova fiber]

fun bug of the day, not mine this time

my mmio tests depend on being able to read shared memory inside a signal handler

this works fine on linux

but enabling zram when building linux breaks this

so i think zram doesnt handle reading from a faulted page inside signal handlers properly

it always returns 0xFFFF'FFFF from a read

[nova fiber]

moving to the new system and its going about as badly as i had expected

on the plus side i no longer burn cpu time when idle

so its a partial success

previously -smp N would control how many of those cores would be pinned at 100%

also really exposes how shit linux handles complex cpu topology because the 4 cores would always be the furthest apart on the die

[nova fiber]

success

now for the rtld

[nova fiber]

it would seem i cannot escape needing a bytecode vm to deal with firmware

damnit

obp needs a forth interpreter

[nova fiber]

perhaps the most heinous thing i've done so far

[urban schooner]

usually both yield and normal reschedule hit the same register save/restore path

[nova fiber]

previously i was just flinging a self-IPI to yield a thread but that is apparently not good

[nova fiber]

openfirmware only boots aout files

i will have to somehow bludgeon my build process into generating an aout file

[nova fiber]

i have discovered that this is actually a useful pattern

i can use setjmp to communicate the result of a handle wait

[nova fiber]

mmm im starting to see why posix doesnt let you mmap a process

quite a few edge cases

on the plus side ipc, rtld, and shmem is much easier to do in userspace

no memfd_create and sharing fds needed, just handing virtual addresses around is great

[nova fiber]

wow this actually makes things so much easier why does no mainstream os do it this way

[tropic berry]

do what?

[nova fiber]

allow mmap to use a pid, so you can map another processes address space into your own

[tropic berry]

huh

i see

[umbral grotto]

that seems interesting

the only problem i guess is that you don't know what src address to map

[nova fiber]

theres still pipes to communicate stuff like that, but for program loading its so much nicer

nearly all my syscalls additionally take a process handle so (nearly) all syscalls can be invoked in a process context

makes program loading really easy, i just invoke mmap in the target process from the current process

since process creation doesnt create any threads or do any file io its fine to do all the work non-atomically in the target process then spawn its first thread once the entire environment is setup

added benefit that i can have coff, pe32, or whatever program format i want and its all done in userspace

the kernel doesnt care or even know about how programs are loaded beyond the init process

[umbral grotto]

oh i thought you meant like map a part of another process' address space into your own

so you can access it directly

not have a process modify another's address space

[nova fiber]

Yes that as well

You can do both

[umbral grotto]

ah

i have only one of these currently but i feel i may need to do both, or have a syscall that allows reading/writing into another process' memory

[urban schooner]

i guess thats just not the unix way

the unix way is abusing fork

[nova fiber]

windows has ReadProcessMemory and WriteProcessMemory but they're a real pain in the ass to use

[umbral grotto]

why

[nova fiber]

using any kind of data structure becomes a soup of those calls

also slow because each time its a syscall

following a linked list goes from cpp head = ptr; while (head != nullptr) { head = head->next; } to ```cpp
Node head;
if (!SUCCEEDED(ReadProcessMemory(guestHead, &head, sizeof(head))) {
abort();
}

while (head.next != nullptr) {
if (!SUCCEEDED(ReadProcessMemory(head.next, &head, sizeof(head))) {
abort();
}
}

[nova fiber]

once again back in userspace