Boron (not vibecoded)

[glass heart]

Suppose the fixed view size is 256k. In this case there would be 8 views created for the 2MB read and 1 view for the 128B read

The 8 views for the 2MB read would eventually be filled in because all the 2 MB are read from however the 128 byte read would only have one page filled in out of the full 256K view

When a view is instantiated it's as if you call mmap one time

[crude arrow]

Does this fixation provide any benefit in comparison with a more flexible, say page multiple approach?

[glass heart]

Honestly I have no idea

Supposing that you'd allocate views based on the read size with a safe minimum view size of 256K

Then you'd have two views, a 2MB one and a 256K one

But if a view smaller than 256K needs to be allocated and there is no room you're probably going to have to unmap the 2MB view

Additionally it would involve more code to handle reading from these mixed size views

And you don't have as many options for data structures to access those views

With fixed size views you might be able to use a hash table, or a radix tree, or something, but with mixed size views you might be stuck using a binary search tree (AVL, RB etc)

[crude arrow]

but is it that bad? AVL or RB trees? It's the same VA bookkeeping as with VADs, which is already there.

[glass heart]

Yes, that's the point

With fixed size views you have access to a wider gamut of data structures to make the file offset -> view base link

And it allows you to catch the case where there is no such view faster

[glass heart]

so i think i can finally start working on the view cache management itself

and my copy-read/copy-write/MDL-read/MDL-write primitives

tomorrow though

and then i can finally get started on the fucking ext2 file system driver

[glass heart]

ok so i implemented my copy-read and MDL-read primitives

testing them it seems to just work

[glass heart]

i am starting work on the ext2 file system

i will get something to show off soon

[visual ledge]

oh wow its your os

[glass heart]

yeah

thats my os

[visual ledge]

crazy

[glass heart]

taking a little break because i need to work on my other "dm" project (something big is brewing and i'm preparing)

you'll see..

[glass heart]

okay, so i'm back from my break

i'm hoping that tomorrow is the day i will read the root directory inode

[glass heart]

today i managed to finally get ext2fs.sys to read my partition's data, i now need to implement the inode read code

[glass heart]

comparison between nanoshell and boron

nanoshell:

// (note: block groups are preloaded in nanoshell, cached in boron)

void Ext2ReadInodeMetaData(Ext2FileSystem* pFS, uint32_t inodeNo, Ext2Inode* pOutputInode)
{
    ASSERT(inodeNo != 0 && "The inode number may not be zero, something is definitely wrong!");
    
    // Determine which block group the inode belongs to.
    int inodesPerGroup = pFS->m_superBlock.m_inodesPerGroup;
    
    // Get the block group this inode is a part of.
    uint32_t blockGroup = (inodeNo - 1) / inodesPerGroup;
    
    // Get the block group's inode table address.
    uint32_t inodeTableAddr = pFS->m_pBlockGroups[blockGroup].m_startBlockAddrInodeTable;
    
    // This is the index inside that table.
    uint32_t index = (inodeNo - 1) % inodesPerGroup;
    
    // Determine which block contains the inode.
    uint32_t thing = index * pFS->m_inodeSize;
    uint32_t blockInodeIsIn = thing / pFS->m_blockSize;
    uint32_t blockInodeOffs = thing % pFS->m_blockSize;
    
    uint8_t bytes[pFS->m_blockSize];
    ASSERT(Ext2ReadBlocks(pFS, inodeTableAddr + blockInodeIsIn, 1, bytes) == DEVERR_SUCCESS);
    
    Ext2Inode* pInode = (Ext2Inode*)(bytes + blockInodeOffs);
    
    *pOutputInode = *pInode;
}```

boron:```c
BSTATUS Ext2ReadBlockGroupDescriptor(PEXT2_FILE_SYSTEM FileSystem, uint32_t BlockGroup, PEXT2_BLOCK_GROUP_DESCRIPTOR Descriptor)
{
// If the block's size is 1024, then the block group descriptor table
// is at block 2, otherwise it's bigger, so at block 1.
uint64_t Address = FileSystem->BlockSize == 1024 ? 2048 : 1024;

Address += BlockGroup * sizeof(EXT2_BLOCK_GROUP_DESCRIPTOR);

return CcReadFileCopy(
    FileSystem->File,
    Address,
    Descriptor,
    sizeof *Descriptor
);

}

BSTATUS Ext2ReadInode(PEXT2_FILE_SYSTEM FileSystem, PFCB Fcb)
{
EXT2_BLOCK_GROUP_DESCRIPTOR Descriptor;
BSTATUS Status;
PREP_EXT;

uint32_t InodeNo = Ext->InodeNumber - 1;

// Determine which block group this inode belongs to.
uint32_t InodesPerGroup = FileSystem->SuperBlock.InodesPerGroup;
uint32_t BlockGroup = InodeNo / InodesPerGroup;

Status = Ext2ReadBlockGroupDescriptor(FileSystem, BlockGroup, &Descriptor);
if (FAILED(Status))
    return Status;

// Get the block group's inode table address and the
// inode's index within that block group table.
uint32_t InodeTableAddr  = Descriptor.InodeTableBlockId;
uint32_t InodeTableIndex = InodeNo % InodesPerGroup;

// And finally, read.
uint64_t Address = InodeTableAddr + InodeTableIndex * FileSystem->InodeSize;
return CcReadFileCopy(
    FileSystem->File,
    Address,
    &Ext->Inode,
    sizeof(EXT2_INODE)
);

}```

first inode metadata read

[glass heart]

and here is the actual data

[fast prism]

Very nice

[glass heart]

but i just realised something

why in the hell am I using the cache to read from a file

the page cache of the partition object is only supposed to be used for metadata

WTF!

[glass heart]

and here it is

a full directory listing

[echo saffron]

@random sleet u gotta catch up

[glass heart]

wdym

whats he doing

[echo saffron]

inside joke

he's been doing vfs for a week now

trying to get it right and all lmao

it's almost done tho 🔥

[glass heart]

the joke is on me, I've been doing it for like a year and a half (on and off with a massive break last fall & winter)

[glass heart]

oh it almost just worked yesterday

now it really works

(reading from offset 4096 in the file)

[glass heart]

also i made a change which makes things considerably faster*

*probably

which is that there is no longer a central mutex for the object namespace (finally)

it's per-directory now

meaning concurrent lookups can be performed at once

[glass heart]

it's crumbling

[mossy thicket]

i disagree with the backslash path larp

[glass heart]

Ok

[mossy thicket]

reason i think that is

even they wouldnt ahve done that if they had a choice

[glass heart]

I was thinking to use slashes too at some point because I want to create a userspace distro using mlibc and most programs use "/"

It'll be annoying to convert paths and so I'll end up having to switch

[glass heart]

ooh, wdym?

were they forced by something?

win32/DOS?

[mossy thicket]

os/2 and by extension, dos

[glass heart]

that makes sense

well seeing as I am not looking to simulate win32 right now I think I will swap the separators

[glass heart]

strange cutoff starting with offset 12k

oh wait!

thats when the first indirect block starts to kick in

oh yeah it looks like i just flipped a condition (if the singly indirect block pointer is zero, then read zeroes, but the condition was mistakenly the inverse)

it works now

now i need to start making sure references arent leaking and things

[glass heart]

libboron.so running from ext2

[primal gorge]

thats nice to see it all come together

[glass heart]

indeed it is!

i still need to implement the write part of the I/O machinery

but i'm happy to see it coming together

[noble sphinx]

nice

[chrome tinsel]

_ _

[glass heart]

yesterday i have made the first user application

but i can't actually run it

because i need to implement libboron's elf loader

[crude arrow]

@glass heart is this that fireworks that some people ported to their OSs and use as a test? I tried to search in the boron repository, but didn't find (from the phone it's not so easy)

https://github.com/iProgramMC/NanoShellOS/tree/master/apps/Fireworks

[glass heart]

Well that's the original version from 2022 which doesn't really act like a stress test

https://github.com/iProgramMC/Boron/blob/master/drivers/test/source/fworktst.c This is the Fireworks test (2023) from Boron which is the multithreaded stress test

[glass heart]

aight

so i'm close to actually running an executable from libboron

what i need to do is implement dynamic linking (so, parsing the export tables of every library, etc)

implement a global heap that libboron uses for its own dynamic linker (the heap itself is done, i need to expose it in a global way now)

and jump to the executable

then i wanna implement support additional libraries as well

idk if i mentioned this before (i probably have), but i want boron to have two ecosystems: a native boron one which will be my own creation, and an mlibc based one

and i plan to actually have both run at the same time kinda

my plan is that libboron loads boron-native applications and mlibc's ld.so loads mlibc applications

so i can implement things like the window manager in my own ecosystem and then mlibc can run alongside boron applications

kinda like the posix subsystem of old NT

and the interpreter path will be used to determine whether to load libboron.so or mlibc ld.so

[primal gorge]

exciting! I'll be interested to hear how you go with this, I'm planning to do something similar.

so libboron is a vdso + dynamic linker of sorts? nevermind, found where you're explaining it

[glass heart]

Not a vDSO, because I'm not exporting parts of the kernel for userspace access, but a dynamic linker yes. And a system call caller too

Kinda like NTDLL

In fact, exactly like NTDLL actually

But honestly I'm thinking of doing this in parallel with mlibc's own ld.so because it's already a thing and it's already good

[primal gorge]

makes sense, you get the best of both worlds.

[glass heart]

To start on the mlibc ecosystem though I will probably have to switch to linux because I'm not sure WSL1 can handle all the crap that's being pulled

Also as a stretch goal in the future, possibly when I have the initial command line shell running, or later when I get an actual GUI, I want to port this OS to the Wii

[primal gorge]

that'd be cool

[glass heart]

Now reading the libraries' path

Currently init.exe only needs libboron.so so I can skip loading other dlls for now

I just need to resolve the relocations and then I should be able to jump to the entry point of init.exe for the first time

Once that happens, I think I'll start on a PTY device type thing

[glass heart]

well it looks like i wasnt really doing things wrong after all

i was parsing the section headers for the symbol table, string table, and GOT and GOTPLT sections

the GOT and GOTPLT sections were useless leftovers since i already ended up doing the refactor i mentioned LOL

and the symbol table and string table are used for debugging in the crash handler

so we good

[glass heart]

It worked

Init.exe is running

Now I should work on importing shared libraries other than libboron.so

Also now we are really using the ELF hash table instead of doing arcane hacks to get the size of the symbol table

Cleaned up logs

and that's about how far it went before kicking the bucket

stack trace in here

and interestingly before it ends up here it is supposed to crash ALREADY

because it says "Declaring access violation on VA %p because I don't know how to handle such a write fault"

(ie. unimplemented case)

maybe the PTE had it already set and its a nothing burger

that address is hella suspect tho

[glass heart]

so why was it trying to write to that? well, because MmProbeAddressSub actually checks the wrong thing!!!

[glass heart]

it works now

[glass heart]

Future plans once I have a workable OS

OS Ports:

• 32-bit x86 port for legacy hardware (i486 class machines, possibly even i386 if I feel like it)
• PowerPC port for Wii
• AArch64 port for Raspberry Pi 3B+
• Possibly an arm32 port for iPhone (I have an iPhone 3G I can hack on)?!

Ecosystem:

• NanoShell applications running natively on Boron x86_64 and x86

[wild widget]

I wouldn't recommend i386, it's extremely annoying to support due to (mainly) its lack of cmpxchg and kernel writes ignoring the W bit in PTEs

So a copy_to_user equivalent on i386 needs to manually check PTE permission bits for each page it touches

Not to mention that it makes it impossible to support CoW kernel memory

[vestal river]

you can change that in cr0 (the WP bit iirc)

[wild widget]

That was introduced on i486

On i386 it doesn't exist

[vestal river]

ah

i assumed i386 => 32-bit x86

not actually i386

[noble sphinx]

And ig you must grab the address space lock also?

[wild widget]

Yeah i486+ is fine, that has all the core primitives you need for a modern OS

Mainly proper page permissions that apply regardless of privilege level and pointer-sized cmpxchg

[wild widget]

Not really, hw doesn't do that and you have to make sure that still works so as long as you do the same PTE lookup process as hw except manual you're fine

Actually no because that's not atomic in respect to tlb flushes

Oh yeah that's another thing i386 doesn't have: invlpg. The only way to flush tlb on i386 is a cr3 write and thus you cannot flush specific pages, only clear the tlb

[vestal river]

does it at least have the global bit?

to keep the kernel around

[wild widget]

Nope

[vestal river]

actually makes sense, how would you evict global pages without invlpg

[wild widget]

But tbf i486 also doesn't have global

Don't remember whether it was introduced with i586 or i686 but definitely not with i486

[vestal river]

huh

[noble sphinx]

Hw would generate a page fault if permissions drop in the middle, whereas your memcpy wouldnt right

[wild widget]

Yeah that's what I realized right after sending that

You'd have to raise IRQL or take an address space lock

And taking an address space lock would most likely involve raising IRQL since that's required to do a cmpxchg operation on i386

[echo saffron]

the page tables can be modified from another cpu

[wild widget]

i386 doesn't have SMP

[echo saffron]

oh

lmao

real

[wild widget]

Well it technically has the hardware facilities to handle multiprocessor systems but I'm not aware of any and I sincerely doubt they exist because on an MP i386 system it's completely impossible to implement cmpxchg

On a UP system you can at least raise IRQL and then do the operation but on an MP system that doesn't work either

Also fun fact this means that on i386 UP systems cmpxchg in user programs involves a syscall

[glass heart]

Yes those are the quirks I'm thinking of when deciding whether I want to support i386 or not

It's pretty important to not ignore the W bit even in ring 0, but apparently intel didn't get the memo until the 486

(Though I can't blame them, the i386 released for the first time in 1985)

[wild widget]

The most annoying quirk imo is the lack of cmpxchg

Because it leaks all the way into userspace as well, makes SMP impossible, and (if you want to be able to support both i386 and i486+ in the same build) turns a normally fairly cheap operation into an indirect call at best

[glass heart]

Now for the normal plans to implement:

• ✅ thread killing -- threads can only be outright killed while they are in userspace, and if a thread is killed while waiting in kernel mode, it reacts the same as if it receives an APC (which are currently unused), meaning that alertable waits are interrupted with the status code STATUS_KILLED

• ✅ OSExitProcess - a way to exit the process AND all active threads

• thread local storage using FS or GS base (not sure which)

• ✅ add IO infrastructure for writing files including modified page writer

• ✅ add OSWriteFile system call

• ☑️ add OSDeviceIOControl system call

• ☑️ a keyboard device object openable with OSOpenFile, and they would be accessible to userspace (originally said keyboard AND mouse but I didn't implement mouse support)

• ☑️ a pseudoterminal device which can have several IO controls associated with it such as "make active"

• ☑️ handle inheritance for processes, handles are inherited in the same slots so that the parent and child can easily know what is what

• ☑️ a basic shell

• ☑️ a terminal/session manager in user mode so that that shell can be connected to a pseudoterminal device

• the ability to switch between active terminal sessions

• ☑️ support in OSDLL for environment variables

• ☑️ the inheritance of environment variables from parent to child

• ☑️ the ability for the init process to read environment variables from a config file on disk or in RAM

• ☑️ add directory related system calls
  ...

(☑️ - updated after 26/02/2026)

There's way more after this but this is what's near

[echo saffron]

whyc is the prefix OS instead of Os

[glass heart]

this is not Osmium OS

if I really wanted to use the name of the OS as a prefix I'd do B, Br, Brn, Bor, or Boron

[echo saffron]

yeah but does your memory manager use MM as a prefix?

[glass heart]

No

[echo saffron]

exactly

why uppercase this prefix

and not the others

[glass heart]

the second reason is because the official Wii SDK uses OS prefix

and I liked that

[echo saffron]

interesting

but that inconsistency is killing me

but it's good that there is a reason and i can respect that

[glass heart]

true, my io system is not prefixed with IO but with Io

will's mintia2 is more consistent because it goes with Os

and mintia1 is also slightly more consistent because it uses OS but also IO

why does IDA think boron might be an SNES ROM

[vestal river]

lmao

i wonder how ida handles rep/sep #$30

[noble sphinx]

Nes port when

[vestal river]

and the m/x flags in general

[glass heart]

gimme a second i actually have a snes rom with source code

for this snippet ```
reset:
sei
cld
clc
xce

i16
ldx #$1FF
txs

pea $0000
pea $0000
pld
plb
plb

ai16

; zero the CPU registers nmitimen through memsel
; note: since A is 16 bits there are 2 bytes written
stz nmitimen; and wrio

these macros```
.macro ai16
rep #$30
.a16
.i16
.endmacro

.macro i16
rep #$10
.i16
.endmacro

this is what it looks like

[vestal river]

i mean yeah but how does it figure out the state of m/x on subroutine entry

[glass heart]

i dont think it does really

or im not sure

[vestal river]

if the subroutine is for example called from an irq handler where the state cannot be inferred from the caller

i ran into the same thing in radare2

while working on my project

[glass heart]

what's radare2?

[vestal river]

https://en.wikipedia.org/wiki/Radare2

i was using it as a 65c816 disassembler that knows how to read elf files :^)

[glass heart]

i dont think IDA really knows for sure

[glass heart]

ah yes right

its supposed to because the instruction encoding differs

[vestal river]

yeah instruction operand width changes

which is why it actually matters

[glass heart]

like LDX #$1000 is encoded as A2 00 10 which in I8 mode would mean LDX #$00, and whatever opcode 0x10 is

you might need to supply it hints manually

[vestal river]

also as a side note, check this out ```
qookie@selenium ~/projects/paaliaq λ file build/fw.p/boot0.elf
build/fw.p/boot0.elf: ELF 32-bit LSB executable, WDC 65816/65C816, version 1 (SYSV), no program header, not stripped

the 65c816 has an officially assigned elf e_machine value

the 6502 does too

[glass heart]

lol really

which one

where can i find a complete list of elf e_machine values

thats hilarious

@vestal riveroh i bet thats for the recent llvm-mos toolkit

oh i found this https://gabi.xinuos.com/elf/a-emachine.html

[vestal river]

it was assigned a while ago iirc

ah i found the post requesting it

it's from 2020

https://groups.google.com/g/generic-abi/c/qaPzp2lRzDA?pli=1

and it seems to be from llvm-mos? or some similar project also using llvm at least

[glass heart]

.

[primal gorge]

Do you want that pinned?

[glass heart]

No need, thanks

[glass heart]

boron is currently 43K lines of code

specifically:

boron/ - 29622
borondll/ - 1225
common/ - 4105
drivers/ - 11931
user/ - 494

total: 47377

command: find -name '*.c' -or -name '*.h' -or -name '*.asm' | xargs wc -l

comparatively NANOSHELL:
apps/ [excluded DOOM and TCC] - 14805
crt/ - 7718
include/ - 7579
src/ - 56165
total: 86267

[noble sphinx]

just cloc --vcs=git .

[glass heart]

Well I had to get a full sum including blanks and comments, and I also had to manipulate it a bit to remove big font data headers (in the case of nanoshell), and build artifacts

[noble sphinx]

cloc prints both and allows to exclude files also iirc

[glass heart]

anyway, nanoshell src/ is 42288 sloc which means the code is 75% dense

and boron boron/ is 15276 sloc which means the code is 51% dense

surprising

https://github.com/erikbern/git-of-theseus

amazing how much code added in 2023 and 2024 actually survived

but also not that unexpected

[mossy thicket]

im analyzing my thing with that and its taking forever. cant wait to wait like 5 min just for it to spit out a completely flat graph because it didnt consider anything in a .df file as code

Traceback (most recent call last):
  File "/Users//Library/Python/3.9/lib/python/site-packages/git_of_theseus/stack_plot.py", line 24, in <module>
    from .utils import generate_n_colors
ImportError: attempted relative import with no known parent package

great

[placid sail]

huh neat tool

lmao

[primal gorge]

Lmao, is that because of a library you pulled in?

[glass heart]

okay

so i just pushed the commit dealing with thread termination

it now pretty much works but there are some caveats

if the thread owns any mutexes (what NT would call mutants), then the pointers remain in memory and remain stale, which I will need to fix

and also the thread's stack isn't being freed at all by anyone when a thread exits

i'm thinking of either relegating this job to OSDLL (managing user thread stacks), or to the kernel

most likely will go for the former

[glass heart]

I needed thread termination so I could implement OSExitProcess in a satisfactory way

though that makes me realize

I should probably add some support for exit codes

[glass heart]

Things That Use The HHDM In Boron:

• HAL: PCI MSI-X interrupt set (include/hal/pci.h)

• KE: SMP init (to create the processor list) (ke/smp.c)

• LDR: Loader file page reclamation (ldr/loader.c)

• MM: Page table manipulation (mm/amd64/pt.c)

• MM: Page cache manipulation (mm/cache.c)

• MM: Page fault handling for COW duplication (mm/fault.c)

• MM: MDL Memory Copying (mm/mdl.c)

• MM: PMM Page Frame Database construction (mm/pmm.c)

• MM: Pool Header management (mm/poolhdr.c)

• MM: Debug: ensuring user half is empty on address space teardown (mm/teardown.c)

• HAL: Browsing the ACPI table (halx86/source/acpi.c)

• HAL: Accessing APIC registers (halx86/source/apic.c)

• HAL: Accessing I/O APIC registers (halx86/source/ioapic.c)

• NVME: Access to pages provided while setting up (stornvme/source/commands.c, ./enum.c, ./rdwr.c)

• TEST: Mm1 performs CoW test (test/source/mm1tst.c)

i'll probably need this later so i can perform a refactor when i start porting this OS

[glass heart]

also this TODO

#arm message

i kinda lost the pi 3b+

and its post-mid 2025

[glass heart]

😬

[glass heart]

but why did I add this?!

it could be locked for a legitimate reason

[glass heart]

btw i think i fixed it

the rwlock issue i could just ignore cause it doesnt mean anything if you catch the spinlock being locked on a multicore system goddamn it

only on a single core system

and the timer issue, that's the sleep timer of a certain thread that's being fauled on

[glass heart]

small step towards the write ecosystem working

i now allow pages mapped with PAGE_WRITE permissions to be written to

huge, i know

no modified page writer or lazy writer threads yet

[glass heart]

i dont think i fixed that bug after all

seems to happen more frequently on low memory?!

nah

[near panther]

did you initialize it right?

does this happen only with paging?

[glass heart]

I'm not sure what you're saying

[near panther]

oh

did you initialize paging right

[glass heart]

of course ?

like i did that 2 years ago

[near panther]

idk then

[glass heart]

from what I can tell there is a timer hanging from a deceased thread that sticks around for some reason

and by deceased thread I mean one that exited and doesn't exist anymore and is freed

[near panther]

i doubt a timer triggers page faults

[near panther]

oh

ok maybe that could trigger a page fault

[glass heart]

if the timer was once available, and it was added to the timer queue, and then the memory containing it got freed, then of course it can cause a page fault

[near panther]

right

actually yeah

[glass heart]

Now

There are two offsets at which this timer could possibly be

because of course the thread is freed

or it could be a stack allocated timer in which case... fuck

[near panther]

rip

[glass heart]

well

a super unlikely crash happened

some fucking idiot jumped to NULL

[glass heart]

Well

Let's unveil the cause shall we?

Of course its OSSleep

What

Right

I dont remove the thread from the fucking timer list when I cancel the timer

Bummer, I thought I finally caught it

????

Why the fuck does this still appear but my debug code doesnt trigger??

KepCleanUpThread(ffffa080020c64a0)
KepCleanUpThread(ffffa080030f00a0)
KepCleanUpThread(ffffa080020c68a0)
KepCleanUpThread(ffffa080020c60a0)
KepCleanUpThread(ffffa080000738a0)

*** STOP (CPU 1):  invalid page fault thread=ffffa08000089978, ip=ffffffff8001c456, faultaddr=ffffa080020c6a48, faultmode=0000000000000000, status=5

i think the faultaddr is inside of ffffa080020c68a0

offset 0x1A8

WaitTimer is at offset 0x188

as such, the fault is inside KTHREAD::WaitTimer.EntryTree.rbe_link[1]

i think this might be an SMP issue

i dont think i get the issue anymore

with this latest bugfix

where i basically just track which queue a timer is enqueued to instead of assuming it'd be the processor it was initially enqueued on's timer queue

[glass heart]

so in terms of the modified page writer thing

first i will have to make it so that the PFN knows what FCB and offset it is a part of

which i can do, while still fitting the data in 32 bytes (on 64-bit)

my proposal is to literally chop off the last 16 bits of the address of the prototype PTE and the fcb pointer

which would save 32 bits

plus 20 spare bits out of currently 26 would net me a total of 52 bits

i could actually chop off some more because the page cache doesn't yet support mapping all 2^52 pages

goes up to 256TB which is 48 bits

[mossy thicket]

Why can you do this

[glass heart]

x86_64 by default only uses the lower 48 bits of the address for actual addressing, so just OR 0xFFFF000000000000 and you're good to go

now, if I were to do LA57, then I could still just hardcode the upper bits themselves because the VA space where FCBs and proto PTEs are allocated is limited

on 32-bit this is actually not possible so i will have to just ball out

[mossy thicket]

Makes sense

[glass heart]

hey @shadow lynx can you pin this one please

[glass heart]

and this

[glass heart]

i said no previously but i kinda figured out i would need these

thank you

[shadow lynx]

np

[glass heart]

            // As of this point, this PFDBE has a reference count of zero, its type is PF_TYPE_USED,
            // and the only reference to it is a weak one in the page cache. Now, since we have
            // unlocked the page frame database lock, additional references may be added to this page.
            // This doesn't matter.
            //
            // For example, when a program throws a page fault, and this page is brought into use,
            // its Modified flag will be kept as one, its reference count will be increased, and it
            // won't be added to the modified page list. If it's fully unmapped again by the time we
            // finish this IO operation, then the page will be re-added to the modified page list.
            //
            // However, even after all of that, if we attempt to call MiTransformPageToStandbyPfn,
            // STILL, nothing will happen, because the page was modified *after* we started the IO
            // process.  At worst, data that is partially written is written to disk for a bit, and
            // then the actual data that was meant to be written, is written.```

yap yap yap

[random sleet]

holy yap

[glass heart]

maybe i should try to compress this down into basically "this shit is safe because regardless of what happens to the page frame after we removed it from the modified page queue, it can be used in whatever way, because the watchful eye of the page fault handler will not let modified pages end up in the standby list without being written"

[glass heart]

but i guess this helps me think

😄

[glass heart]

okay i wrote it

now i need to make it initialize, and signal the event when something's up

[primal gorge]

imo its fine, sometimes there is a lot to say. FWIW I would be quite happy to come across a comment like that, since it gives you a summary of some of the system design.

[glass heart]

Well thats a darn good sign

The modified page writer is trying to write the data from modified pages into their backing store (in this case, I opened the file path \InitRoot\test.sys, which is an init root file and cannot be written to)

It doesn't succeed of course ... but give it time and it'll start to succeed

right now the MPW just emplaces the PFN that failed to be written back onto the queue of pages to write

[glass heart]

im gonna mess with the nvme image i have mounted

well

i am

well the write did go through

but for some reason the page is added back onto the modified list

or erroneously reported as such :/

no its actually added, seems like the page is freed when it goes through the MDL, and when it's freed, it's added back because I forget to clear the Modified flag

[glass heart]

Update. I think I have implemented most of the file write infrastructure. I also just added the OSWriteFile systemcall.

[glass heart]

right now I'm thinking about fork

there are some... shall we say... Issues

In terms of duplicating the handle table, there isn't an issue really

The issue is with memory

I don't know of any other memory kinds I'd need to support so here are the cases I know I will need to support:

• Read-only or read-write file mappings (not CoW)
• Anonymous mappings
• Copy-on-write file mappings that have had pages written to

The first one, duplication is trivial

For the second one, I'd basically have to make some kind of section out of each VAD that includes all of the physical pages so that they can be faulted in

I think this is called an "amap" in unix/linux parlance?

For the third one... honestly, I'd do a similar solution called a file section/file amap which, when faulted on, and where there is no page present, you load it from a file instead of creating a zero filled page.

These amaps would be refcounted but they'd be entirely cloned if someone forks after they've already been forked into existence

When you unmap these amaps the refcount decreases and everything goes away if the refcount becomes zero.

[glass heart]

actually i think the pseudoterminals should be implemented in user space

[glass heart]

also im probably going to need a FIFO object and a way to open handles on behalf of another process (handle transfer)

[idle urchin]

yoo boron progress ?

yooo

[glass heart]

boron's progressing, yeah

i just need to figure out what's next, 's all

[idle urchin]

very cool

thats understandable

[glass heart]

It works in vmware seemingly so thats good

[idle urchin]

yea thats nice

[glass heart]

• fifo object
• counter dispatch object, signalled as long as its value is non zero, decrements automatically on acquisition
• conditional variable dispatch objects?! or could they be implemented in userspace maybe? pthread cond vars have a special pthread_cond_wait which atomically unlocks/relocks a mutex when waiting on a condvar

[meager widget]

what for do you need conditional variable dispatch objects?

[glass heart]

i think i wanted them for my fifo objects

i have a ring buffer implementation that uses condvars for the empty and not empty signals

and additionally i'm not sure how i could implement those in userspace

[glass heart]

fluff uncovered?! turns out this 164 loc file is actually 52loc

😭

[echo saffron]

You can actually make it 4 lines of code

[glass heart]

ik you can smash all 49 loc that arent preprocessor statements into one but thats not what i mean

[echo saffron]

i mean, you did get rid of the comments and most whitespace :^)

[glass heart]

anyway i do prefer having the code split up with empty lines

makes it easier on the eyes

consequently I don't like codebases where there are giant clusters of lines that do multiple things when they could be split up

[idle urchin]

funny

[glass heart]

it turns out the least amount of memory that boron will run under, after I've fixed a bug regarding page caches, is 4032KB

that crash is intended because that test really is trying to access an invalid path

4024K**

Without any drivers (except HAL of course), init runs successfully in 3652K

[glass heart]

everything is loaded here except the test.sys driver

[glass heart]

and here only kernel, hal, libboron and init are loaded

[glass heart]

oh btw

boron is 2 years old at this point

i know i'm late but uh.. yeah

19 August 2023 is the first commit

well, in the repo, it is 20th August 2023, but in NanoShell64:rewrite it's the 19th

[amber dune]

Happy late boron day

[glass heart]

i have implemented FIFO pipes in boron

i also plan to implement named pipes

then I will probably start building new userspace programs such as a terminal emulator

honestly, I am considering just implementing a PTY device in the kernel and having syscalls for this PTY device + io controls would be redirected to those

[glass heart]

did a bit more work today

i reorganized my userspace makefile, everything is now under user/ and there is an actual dependency hierarchy

the "Makefile" is the one that calls make -C into every subdirectory in proper order, and "CommonMakefile" is the part of makefile each user program includes

one quirk: it seems I have to append .j to the actual rules because I'm having a skill issue and it doesn't work without it

nevermind i can make it simpler

[glass heart]

now using semantic versioning. it works like this:

• major and minor build numbers are incremented manually (though the major version will probably remain 1)
• the patch/build number is incremented every time the kernel is linked

[finite flint]

what's with the .j extension? AFAICT it's just a dummy to not collide with the name of the application directory, in which case why not do

.PHONY: FORCE
FORCE:

%.j: FORCE
        @$(MAKE) -C $@ $(MAKECMDGOALS)

also the MAKECMDGOALS thing is problematic as now you cannot make app_a app_b anymore.

[glass heart]

It's a skill issue

.j means .job

# Boron application root makefile

# TODO: Remove the ".j" suffix (stands for job).  But it conflicts with the dir names.

# Add your libraries applications to this list.
# They will be built in this order - make sure to put libraries and
# applications their dependencies!
APPS := \
    libboron \
    init \
    test

### Do not modify below ###

.PHONY: all

all:
    @for app in $(APPS); do \
        $(MAKE) -C $$app $(MAKECMDGOALS); \
    done

# Other goals you might want
clean: all

I ended up doing it manually

and just ordering the hierarchy manually

which sucks. but I couldn't figure out anything better

[finite flint]

oh I didn't notice the

test.j: libboron.j
init.j: libboron.j

stuff at first. that is actually pretty ingenious

[glass heart]

I was hoping it'd work, but nope!

That makefile only compiles libboron and test!

Init is skipped fsr

[finite flint]

why doesn't it work?

it looks valid to me

[glass heart]

I don't know

[glass heart]

okay, the top priority in my opinion should be to launch a new app from init.exe

that means adding all the relevant machinery from within libboron/osdll

Vro in which process' context are you allocating the PEB

Additionally you are already allocating it in initproc.c WHY DOES THIS EXIST

What was I smoking

This is kinda outdated

Also I got rid of this file anyway

Nevermind I was attaching to the process but still

so as it stands the PEB is allocated inside of initproc.c for the initial process

and additionally inside of process.c in OSCreateProcess for some reason

which is really strange

only one of these PEBs is actually used and it's passed into libboron when the process starts

[glass heart]

my debugging tools are not the best but i do have something i can pull out

SYSCALL: (processID) - (syscallNum) (syscallName)

i think i may have left a process attached somehow

and because I already map libboron.so (in the same place), other code ends up executing instead

so like swapping libboron for an incompletely relocated version

[glass heart]

nope it looks like we don't do that badly

however, we do unattach from a process after trying to copy the data into the calling process' address space which doesn't work

OSAllocateVirtualMemory:

oops.

so it does enter the new process now! but it doesnt work because there is no image name & command line

i understand why there wouldn't be a cmdline for test.exe but wheres the imagename

maybe OSWriteVirtualMemory is failing.

[glass heart]

turns out its not

the data written is instead invalid

and here it is. test.exe launched

[glass heart]

38 syscalls, a good start

i've been keeping them alphabetically ordered, but i think this is going to change. until version 0.1 i'm just going to keep appending to the end

on version 0.1 (whenever that is) i'm going to reorder them alphabetically

actually technically that'd be 1.0 but whatever

i predict we're going to reach build number 2000 before we have a fully fledged shell

we're already at 39

there is one problem (at least) though: PspDeleteProcess is not being called!

so there is a reference leak somewhere.

[random sleet]

i sort my syscalls by what they touch

so e.g. all vfs syscalls are grouped together

[glass heart]

i don't really care about the system call numbers personally

i will probably never use them directly

[slow fractal]

My syscalls are just in the order I added them

[echo saffron]

do you dynamically link against the kernel or what lol

[glass heart]

no wdym

[echo saffron]

there needs to be some user->kernel transition and kernel somehow needs to know what syscall was invoked

[glass heart]

thats crazy

no

[echo saffron]

???

[glass heart]

kernel syscalls are exposed via libboron.so which also loads executables and their libraries

[echo saffron]

how does libboron.so know how to syscall, still, there NEEDS to be a user->kernel transition that carries over that information to kernel land

[glass heart]

and also includes some utils such as OSCreateProcess (because the syscall the kernel exposes (called OSCreateProcessInternal) is extremely simple and ONLY creates the process object and nothing else

[glass heart]

user/libboron/source/calls.asm defines the syscall functions which call the syscall instruction

[echo saffron]

so you do eventually hardcode the syscall numbers

or use some enum

[glass heart]

currently they are hardcoded, yes

[echo saffron]

makes sense i guess

[glass heart]

i might later on autogenerate that file based on boron/source/ke/amd64/syscall.asm or whatever

[echo saffron]

honestly i think having a handful of constants is good enough, especially if you can share them with the kernel

auto generating them sounds cool but i don't think it's needed in practice, as you'll ever only increment the number

unless you want an easy way to periodically break all direct system call users lol

[glass heart]

no one will use direct system calls so i dont care

that's in fact why my plan is to link mlibc with libboron.so

[echo saffron]

yeah i know, i'm just having fun, i'm not saying anyone should make direct syscalls

[glass heart]

its kinda like how windows starts by loading ntdll.dll, and then kernel32.dll, msvcrt.dll and your application

in fact processes are started up in a similar way as on windows, as far as i know, wherein when creating the process, only libboron.so (ntdll.dll in windows) is mapped, and that dynamic library itself does all the program loading stuff

[echo saffron]

that's also why i asked how you plan on doing that, because it seemed very similar to windows but you said "no direct syscalls" lol

but there is a direct syscall in libboron.so

[glass heart]

windows also does not technically allow you to do direct syscalls

but it does do direct syscalls in ntdll.dll

[echo saffron]

wdym it doesn't allow direct syscalls

you can

nothing stops you?

[glass heart]

you probably can... but you'll only be able to run your app correctly on anything but the windows version you are targeting

[echo saffron]

exactly

[glass heart]

because the syscalls are sorted alphabetically and keep changing

[echo saffron]

it's not a good design but you can absolutely do it if you wanted to

you can also parse the export table of a certain dll and find syscall numbers at runtime :^)

[glass heart]

i think there were some systems that wanted to prevent direct syscalls

netbsd i think?

[echo saffron]

yeah they have the syscall region thing

where libc does a syscall to restrict from which memory range syscalls can be made

serenity copied that idea

[glass heart]

https://marc.info/?l=openbsd-tech&m=169841790407370&w=2 openbsd apparently

i don't think i will go quite that far

[echo saffron]

oh that i was not aware of

[glass heart]

oh my god they still use CVS??

or RCS

[slow fractal]

wtf serenity doing something non shit for once

[scenic bronze]

i think serenity copied unveil()

but idk about the syscall stuff

anyway unveil is not a good API imo, they shouldve done it capability-based

[primal gorge]

zircon does this I believe, you're supposed to use the vdso to make syscalls

https://fuchsia.dev/fuchsia-src/concepts/kernel/vdso#enforcement

yeah it kind of talks about it here

[glass heart]

boron does not have a vDSO, the closest thing to one is libboron which is a regular DSO

[primal gorge]

ah right, I thought it was a vsdo for some reason

[glass heart]

not yet upstream but i added support for loading shared libraries other than libboron.so. the code was already there for the most part, i just had to add glue code to determine the image base, and to actually find the libraries

and fix a bug in the kernel which caused a VA misalignment offset to be corrected twice in a row

[glass heart]

okay, the changes are now upstream

[glass heart]

i should create some directory system calls

now hmm

there are file directories (in an actual file system), and there are object directories (in the object namespace)

i think i will allow OSOpenFile to open directories and then you can call like OSReadDirectory or whatever to read the directory

but actually i need to think about it

I basically need a way to read a directory's contents sequentially, and a way to get statistics on a file in that directory

(so readdir and stat basically)

but i realized that if we're supposed to have unique file objects for each time the file is opened, then it should be possible to just keep the directory stream pointers in there

[glass heart]

ok, I worked a little bit more on libboron, and now PT_INTERP is parsed correctly and works

and it does allow specification of something other than libboron.so

so helpful for mlibc when I get to porting it

[glass heart]

some logging cleanups done

also i added OSReadDirectoryEntries

which resembles linux's getdents

[glass heart]

im actually not terribly confident with this approach

the problem is twofold:

• I want the kernel to have as little to do as possible regarding bookkeeping, but
• I don't want userspace to be able to manipulate the offset inside a directory in order to cause the kernel to read invalid data

My current approach would also require me to add a OSSeekDirectoryFile which is kinda bad

Then I could permit specifying of offsets but I'd need to verify this offset which could potentially be expensive (e.g. on EXT2)

also 40 system calls now

[finite flint]

If your OSReadDirectoryEntries is similar to getdents64 of Linux, why would you need a OSSeekDirectoryFile? IIRC getdents64 takes the "offset" directly as an argument, couldn't you do the same?

[finite flint]

hm, the way I understood your project, on file exec the kernel first loads libboron.so into memory, then gives it a handle to the executable to be loaded into memory (this is the only way that makes sense in my mind, maybe you do something else?), so the kernel wouldn't see PT_INTERP or would it?

[glass heart]

I have mentioned the problem that I don't want to have to verify a user provided offset, it seems really slow

I could provide the offset+version but the problem is still there

[glass heart]

The initial process is spawned with ONLY libboron.so, yes, so the initial process MUST have PT_INTERP set to libboron.so. But other binaries (created with OSCreateProcess) are loaded as you might expect: the interpreter and the main executable image are loaded, and then the interpreter is jumped to

That's the behavior as of today. Literally yesterday I wrote the code to do that. Before, all programs were loaded using only libboron.so

[finite flint]

I don't think it is THAT slow, maybe it is on ext2 but maybe there are ways to speed it up (ext4 magic, readdir "cookies" or something). Based on Linux to my knowledge not having a "better" interface than getdents64, Linux people probably don't think it is a problem and performance is important to them.

[glass heart]

i know i have the man pages but im not on my pc so here: https://linux.die.net/man/2/getdents64

i dont see an offset here

i guess it uses the same offset as read and write or whatever

[finite flint]

oh what, I could've sworn I've seen something similar but with an offset

[finite flint]

yeah it probably overloads seek

[glass heart]

so this is because I don't have an OSSeekFile

stream pointer incrementation is only done in user space

OSReadFile and OSWriteFile take a file offset

[finite flint]

anyway you could do something neat if you think that performance will be an issue:

• observe that most programs which use readdir will sequentially read all entries of a directory. this translates into OSReadDirectoryEntries on the "offset" that was left by the previous call to OSReadDirectoryEntries.
• what this means is that you can cache only the last "cookie" in a "fast" cache, and have a "slow" cache/some other keep alive to handle the rare case of a program seeking to a different offset.
• because of the sequential behavior of most programs, you will almost always hit the fast cache

what this "slow" cache probably amounts to is snapshotting the entire directory contents which you have to do anyway for conforming readdir I think. I wouldn't be surprised if there are cool ways to do this with B-trees that are zero cost unless concurrent modifications, then logarithmic cost.

[glass heart]

ill pull out of this message the idea to cache two offsets (the offsets before and after the last readdirents call) and mix it with the version idea i had gleaned from linux

i will not be reading all the dirents directly into a "cache", i don't think that's necessary

actually there's already the page cache so even less necessary imo

directory entries are read through the page cache

[finite flint]

no; I agree, and I would be surprised if Linux or any other production kernel does, either.

[glass heart]

I'm pretty sure getdents isn't atomic between calls or anything

so the directory can change between those calls

so why would a "conforming readdir" require such a thing

(or maybe I'm misunderstanding your message)

[finite flint]

yeah, and you have to deal with that somehow (possibly by "pretend it doesn't happen" since it's enough of an edge case to probably not matter)

[finite flint]

IIRC if you've read entry FOO at offset XYZ, if you then seekdir(XYZ) and read there again you should get FOO back.

atleast I have PTSD from reading up on this stuff long ago

[glass heart]

does linux conform to this expectation

[finite flint]

I don't know

ah and I just read the #filesystems channel (I tend to ignore most channels, only occasionally reading them), turns out you've already looked a bit at these things

[glass heart]

I did look into how linux does the rechecking logic for ext2

(it says ext4 there but shh)

[glass heart]

i have changed my age prefix 🔥

[primal gorge]

congrats 🔥

[slow fractal]

Happy birth

[glass heart]

tysm!!

[noble sphinx]

happy cake

[dusk merlin]

birth

[bronze egret]

happy happy happy

[glass heart]

i finally learned about the original architect for this project

his name is Ron, Bo

you can find him at [email protected]

[glass heart]

so i managed to build mlibc for boron, with one caveat: libboron.so is nowhere to be seen

its not being used for syscalls by libc yet and i need to fix that

rtld wont be using libboron for syscalls for obvious reasons

itll have its own copy of important syscalls

[finite flint]

wouldn't it be possible for libboron.so to also be rtld?

[glass heart]

Yes and in fact that's what I would like to do, but I'd need to basically simulate mlibc's rtld in order to make the libc.so work correctly

[random sleet]

vdso support when

[glass heart]

not part of the planned design

[glass heart]

I added a framebuffer driver (framebuf.sys) to boron which exposes the framebuffer as a device file, and couldn't resist doing this (for the record, I am not trans)

[sacred wadi]

nice!

[glass heart]

StorNvme ERROR: Rejecting device 0:14:0 because it doesn't support MSI-X and table size is -1

virtualbox moment

idk why it freezes the initialization procedure though i should look into it

wow

(vmware now)

42 means An I/O error reported by the hardware.

[slow fractal]

Spooky

[glass heart]

qemu-system-x86_64 -bios OVMF-pure-efi.fd -display sdl -m 1G -drive id=nvm,file=stuff.vhd

qemu-system-x86_64 -bios OVMF-pure-efi.fd -display sdl -m 1G -drive id=nvm,file=stuff.vhd -device secondary-vga for the record if I want to test multiple displays

[near panther]

trans

your nvme ruined it tho 😔

also i think that might be the point but this os looks like windows nt

[glass heart]

yep

[near panther]

based choice

now make it run wine

[random sleet]

everything is a file

[glass heart]

Here is that same pattern, this time from a user process!

It does OSDeviceIoControl to get the parameters of the framebuffer, maps it into memory, and writes to it!

[glass heart]

wow

crash that only happens at 320x200 resolution

fixed

[echo saffron]

so why did it happen? :^)

[glass heart]

the last page of the frame buffer was cut off and wasn't registered in the page frame database

[echo saffron]

ah that's pretty simple lol

i was expecting something more fun

[glass heart]

something a bit more fun that i fixed is the fact that MMIO pages didn't get a boost in reference count properly

"why am I tracking references of MMIO pages? not like I ever need to free them" sure, but the BackingMemory function of a file could also apply to regular kernel memory some day

[near panther]

why are you even in 320x200 😔

does limine even support that

[glass heart]

yes

i was messing around

[near panther]

oh

i did 640x480 minimum for mine

[glass heart]

I am thinking that I should probably still implement some kind of terminal object in Boron + some system calls related to it.

It seems like it'd be a bit cleaner

I will still do only the bare minimum in the kernel though

The alternative is to try and simulate it with pipes which might break some programs that expect to be able to write to stdin

[barren ocean]

future N64 support

[glass heart]

buy me an n64 when i start my porting spree and i'll do it /j

and flash cart

[glass heart]

restarting work on porting to x86 (32-bit). it'll pave the way to other 32-bit ports such as ARM, PowerPC, and maybe XR/17032

[glass heart]

these are the symbols I need to define: DbgInit DbgPrintStackTrace DbgPrintString DbgPrintStringLocked KeDescendIntoUserMode KeDisableInterrupts KeGetCPUPointer KeGetCPUPointer KeGetCurrentPageTable KeInitArchUP KeIssueTLBShootDown KeOnUpdateIPL KeRestoreInterrupts KeSetCPUPointer KeSetCurrentPageTable KiBeforeSystemStartup KiDebugPrintLock KiPrintLock KiSetupRegistersThread KiSwitchThreadStackForever MmGetHHDMBase MmIsAddressRangeValid MmProbeAddress MmProbeAddressSubEarlyReturn MmSafeCopy __udivdi3 __umoddi3

[scenic bronze]

why is DbgPrint an architecture thing?

[crude arrow]

https://tenor.com/view/the-office-no-angry-steve-carell-michael-scott-gif-5606969

it'll pave the way to other 32-bit ports such as ARM

[glass heart]

on x86 and x86_64 it prints to port 0xE9

but that wont exist on other platforms

[glass heart]

why⁉️

look at this phone

(not my picture)

wouldnt it be nice to run a custom OS on it

[scenic bronze]

yeah but shouldnt it just be a putchar or something

[glass heart]

right

DbgPrint formats stuff like printf and calls DbgPrintString

so youre right

i should move that part into the arch independent part of the kernel

[scenic bronze]

yeah

the printf should be generic

and the print string or putchar or whatever is arch dependent

or you have some kind of sink system and the architecture code adds its own sink

[glass heart]

debug stuff is supposed to be INSTANT!

[glass heart]

i am like 🤏 this close to getting the 32 bit port linked

just a bit more code which i will write Tomorrow

this DOES NOT mean that the kernel will just work though! i will still need to rewrite all the parts that depend on limine as well as part of the drivers

[barren ocean]

PowerPC 🗣️ 🔥

As long as it supports:

• Device trees
  • some way to load them from something other than the firmware, like being built in, or having some minimal loader with one built in
• PCI (for Xbox 360)
• bare MMIO access from DT (for everything else)
• support for at least ppc750 and ppc970
  I can and will (at least try my best to) port it to all the consoles (GCN, Wii, Wii U, Xbox 360, PS3)

[glass heart]

I don't know what a device tree is

I was going to just write all the drivers and hardcode their MMIO addresses (or at least allow you to specify their base address somehow)

[random sleet]

a flattened tree structure that describes a device

[glass heart]

I don't think the PC or the Nintendo Wii gives me such a thing

[random sleet]

PC will not

but powerpc will

[glass heart]

No, it won't

[random sleet]

why

[glass heart]

Does every homebrew application get a device tree when it boots up?

No

It expects the Wii hardware to be there and to be the same

[random sleet]

linux uses its own device trees

which are passed to the kernel on boot

[glass heart]

they're probably hardcoded at the boot loader level

[random sleet]

depends on the device

sometimes it's passed from the bootloader, sometimes it's from the firmware

https://elixir.bootlin.com/linux/v6.16.9/source/arch/arm64/boot/dts/qcom/x1e78100-lenovo-thinkpad-t14s.dts for example

this laptop has ACPI, but linux uses device trees to boot on there

[glass heart]

I don't think IOS, the system menu, or the homebrew channel produce device trees

yeah that's not the wii

that's something else

[random sleet]

the wii has its own device tree

[barren ocean]

yeah it's internal

you have to include it yourself

[random sleet]

https://elixir.bootlin.com/linux/v6.16.9/source/arch/powerpc/boot/dts/wii.dts

[barren ocean]

nothing will give it to you

[glass heart]

I should add this infrastructure though

[barren ocean]

unlike on OpenFirmware on a Mac, which provides you a device tree

[glass heart]

Well yeah but this one is hardcoded

The bootloader that loads linux doesn't give this to you

[barren ocean]

yes that's the standard for embedded devices

including ARM, even

[glass heart]

It'll be useful when I start porting to other things

[random sleet]

better than hardcoding the arch deps for a single device

[glass heart]

the standard is for the devicetree to be hardcoded?

[barren ocean]

yes

[random sleet]

most sbcs and embedded devices work like this yes

[glass heart]

who said i'd do that

[random sleet]

you're kinda implying that

that's why device trees exist

[glass heart]

when did i ever imply that

[random sleet]

.

[glass heart]

Write all their drivers

Separately

Not hardcoded in the kernel

[barren ocean]

the thing is that several of these devices share similar hardware

like they all have OHCI and EHCI controllers, for example

well, except the GameCube

but you can use (almost - the Wii/Wii U's have some quirks) the same driver across all of them

[glass heart]

so if you load it on the wii it'd load like halgcwiippc.sys, vidgecko.sys, etc

[barren ocean]

it makes no sense to have a seperate driver for each

this is why device trees exist

they describe what hardware is in the device, and where it is

so you can bind drivers appropriately

[glass heart]

that sounds nice

i'll think about it

i'll need to figure out under what namespace (ke, ldr, etc) i'm going to put it in (probably ke), among other things

[uncut lance]

they're only really necessary in the form they've taken for linux if you support all kinds of embedded platforms

[barren ocean]

the discussion above is specifically about supporting PowerPC, where they're the only form of hardware detection really

[uncut lance]

you can get by without them on open and standardised platforms like the PC, for powerpc then you have real open firmware and a true and authentic device tree

[glass heart]

my plan isnt to port to anything other than the Wii

because that's the only PPC hardware I have

and possibly the qemu virt device

[barren ocean]

you can also do Macs easily with QEMU

I also have an abundance of real PPC Mac hardware to test on

[glass heart]

I don't want to do ports to platforms where I don't have real hardware for them

Or, they're a fantasy arch like xr17032 that happens to have a working and good C compiler

[barren ocean]

fair I suppsoe

[glass heart]

Also Boron is in no way usable so I probably shouldn't even be thinking about ports to other architectures

but I have actually tried to write my code in as portable of a way as possible

the biggest hurdle in getting the boron kernel in an Actually Portable state is probably gonna be the 32-bit x86 port

the other ports are going to ride on top of this effort

[uncut lance]

some day i might call you up on that

i would like to try a ppc port some day

after porting mlibc to m68k it turned out to not be too much of a bother so i feel quite positive about trying other ports

[glass heart]

I will also need to move away ALL of the Limine dependencies

I'll probably write a small bit of preloader code that puts all the limine provided information in a struct (for amd64), then do the same for i386 and then the other platforms

[uncut lance]

in the alternative shim the limine protocol for ppc

this is how keyronex/m68k came about

[glass heart]

i could do that

but i kinda wanna be a bit NIH about this

like just create a boot loader information block that the kernel gets on startup and can use

i believe NT does this

i also need to think about what format i should use for device trees and how i'd decode it

there could be a lot of edge cases

[barren ocean]

feel free
I have in terms of PPC Mac hardware:

• iBook G3, 600MHz, 640MB RAM, 20GB HDD, Rage 128
• iBook G4, 1.33GHz, 512MB RAM, 60GB HDD, Radeon 9550 Mobility
• PowerBook G4, 1.67GHz, 2GB RAM, 512GB SSD, Radeon 9700 Mobility
• PowerMac G5, 2.0GHz Dual-Processor, 3GB RAM, 2x128GB SSD, GeForce FX 5200 Ultra
  I also am the primary (currently downstream, though I hope to fix that) maintainer, for, or at least contributor to, Linux on a lot of the consoles:
• maintainer of:
  • GameCube
  • Wii
  • Xbox 360 (at least, the primary maintainer - it's still slightly active from others)
• contributor to:
  • PS3 (helped with distro packaging and testing)
  • Wii U (testing, audio driver, helped a bit with Wi-Fi)
    So I have all of those as well

the only thing I'm missing is a PPC 60x box :P

[glass heart]

also some of the dt information needs to available during the earliest parts of the bringup process as the HAL needs them

[shadow lynx]

Luckily I have a few of those

I should buy a 360 soon

Tbh, I need to get a PS3 and a Wii U too

But I blew a ton of money at VCFMW so I gotta pace it out

[barren ocean]

you should, it's a very funny console

also with BadUpdate, and now ABadAvatar kicking around, you can get into custom code in seconds to at most a minute or so after power on

it's pretty based

it's also litterally all just PCI (no, seriously, it's all just PCI)

and half of the stuff is like, shockingly normal

very odd for a 7th gen console

both the PS3 and Wii (and really all of Nintendo's PPC stuff) was all cursed and just threw MMIO blocks around and you had to know ahead of time where it all is, and none of it was standard even if you did know

on the 360 it's all just PCI devices, and:

• some are completely standard (like USB)
• some are mostly standard parts from SiS (like Ethernet and SATA)
• only of them 1 is weird (GPU)

me and the rest of the Free60 folks keep getting shocked at how much stuff is Just Normal™ and doesn't need a custom driver at all

[shadow lynx]

is it easy to get a framebuffer?

[barren ocean]

uhhhhhh..... no

it's some cursed tiled shit

UART is easy though

or if you just need to see if it's alive you can poke the SMC to set the Ring of Light

[shadow lynx]

oh no

[barren ocean]

yeah

it's like..... kinda documented

in the form of Linux drivers

(one of which doesn't work..... but there's one that does!)

also nobody has figured out GPU acceleration yet
but it's kinda close to an Adreno 200 with the TDP cranked to 200W, actually

Xenos -> AMD Imageon Z430 -> Qualcomm Adreno 200

closer than any Radeon card, ironically

but yeah other than graphics everything else is normal

[glass heart]

the kernel has been linked, some drivers and the entire userspace (which is just a couple exes and sos) also build. what's left:

• a HAL for the legacy PC platform (and probably a port to 32-bit x86 of the original HAL if I want SMP and stuff)
• bootloader abstraction (putting everything limine / multiboot gives me in a new struct that's bootloader agnostic)
• testing!!!!

[glass heart]

first attempt

[primal gorge]

work on 32-bit stuff is exciting to see

I remember when I first got my kernel to link and run (crash and burn? lol) on a 32-bit target, was quite an exciting moment.

[glass heart]

it'll pave the way to other ports such as ARM and PPC

the real hardware devices I have for each:

• iPhone 3G (armv6, not sure how I'll get it to boot but probably OpeniBoot)
• Nintendo Wii (PPC)

[crude arrow]

Got outdated and lost relevance since like 2015. All arm vendors, even such low end (and thus way more affordable for enthusiasts), like rockchip, allwinner, amlogic instantly switched to arm64. So, be prepared to spend a lot of time for porting it and having 0 users as an outcome. After all, no matter how "enthusiast" we consider our projects, it would be sad to know than even in theory a particular port would have no audience at all. Plus, 32 bit arm has an awful VMSA (paging). They screwed it, it is bad, just ewww. Plus firmwares... It's miserable. Forget about UEFI/ACPI. linux oriented device tree halfarsed quasistandard, pulled lamely off Open Firmware is all you would have. But yeah, if you just dream to port boron to that particular iphone, then sure. My response assumed you want a user base. Then supporting 32 bit arm would be comparable with supporting 80286.

[glass heart]

God knows I want to port my OS to the iPhone 3G which was released in 2008, 7 years earlier

I would compare supporting 32-bit ARM with supporting the 80386 instead of the 80286 though since the '286 wasn't even 32-bit

And the 80386 has a ton of quirks

Like the readwrite flag being ignored for kernel mode

And the countless errata in earlier revisions

[glass heart]

I merely want to brag that my OS can run on a phone from 2008. (And a console from 2006, and PCs from the 1990s...)

That's it

[near panther]

power macintosh 6100/60 AV's i486 DOS expansion card

then when you get it running on the powermac's dos card, run it on the actual powermac

and then add support for the dos card

dual boron

[glass heart]

Ok, abstracting away Limine is finished. Now I can get started with the M*ltib**t part

[glass heart]

now it wants a HAL!

I still didnt take EVERYTHING from multiboot, though..

[scenic bronze]

Damn dynamically linked HAL

Full NT LARP mode

[noble sphinx]

nt larp is based tho

[scenic bronze]

Yeah but arguably the dynamically linked HAL is kinda useless

But it's fun so whatever

[noble sphinx]

i guess

[glass heart]

Its useful if you have multiple platforms on the same ISA

[glass heart]

whats next:

• finishing the bootloader conversion for multiboot
• writing the HAL for this platform that uses legacy hardware such as the PIC
• removing MmGetHHDMOffsetAddr from places where the memory address isnt guaranteed to be <256MB, and use something else instead (or, maybe keep using it with my fast 16MB at a time windows that I was thinking about, but make sure there can't be any race conditions or anything)

the plan for this thing is as follows:

• the OS will always map the first 256MB of physical memory
• the rest will be accessible in 16MB windows switched on the fly

[glass heart]

so the HAL is being loaded now

but there is a crash somewhere and i'm not sure where

oh i forgot to initialize the processor list (of which there will only be one for 32-bit x86)

clearly i haven't detailed what i want to achieve with this port.

• port boron to REALLY old computers (so no ACPI, no APM, no MSI-X, etc etc)
• ensure that the entire OS can even be compiled for other architectures ahead of the ports to PPC and ARM
• verify the actual portability of boron in its entirety

[glass heart]

making progress

i REALLY wonder where i'm getting a Double Fault from

wait... hardware interrupt 0x08?

and i see 0x08 and 0x09

if what i read is true, that main PIC interrupts (IRQ 0-7) start at 0x08 and sub PIC interrupts (IRQ 8-15) start at 0x70, then i guess i'm getting PIC interrupts! but too early!!

[echo saffron]

i wanna do an nt larp kernel so bad

[glass heart]

first framebuffer terminal boot

still not complete though!

[glass heart]

so why arent you doing it

anything stopping you?

[noble sphinx]

Laziness I assume

[echo saffron]

the fact i don’t know what i’m doing and that it would probably take a while to plan out so i don’t end up writing slop, and i hate planning :^)

and what infy said i guess lol

[glass heart]

first test executed successfully

looks like i had to update the pool allocator for 32-bit systems

well... almost

[glass heart]

Well, there it is

I fixed all the memory bugs

nevermind

🔥

ok i think i found the cause

nope? fucked something else up?

[glass heart]

turns out the PFN struct has to be a power of 2 because it has to align to a page size

and the code gets mad if a PFDBE (page frame database entry) crosses a page boundary

man this version boots up into the fireworks test so fast

well the amd64 only boots up slower because

• it has to synchronize all the processors
• it has to calibrate the apic and tsc timers

[crude arrow]

Isn't Windows' PFN entry 48 bytes for 64 bits?

[glass heart]

I don't know

But believe it or not, MMPFDBE is 32 bytes on 64-bit in Boron

this comes with a big limitation which is that memory above 16 TiB cannot be managed by the PFDB at all

because the PFN is 32 bits

If it was a full 64, then you'd be able to support all 16 PB that could theoretically be physically addressable

but for now I'm ok with the 16 TB that I have access to

[glass heart]

the first one is done, the second one is I think done as well but necessitates more testing

[glass heart]

firework test still catching bugs in 2025 ❤️‍🩹

[glass heart]

it lasted slightly longer during this specific run, but its still unstable

[glass heart]

I may have found the subtle bug, but I need to run my stresstest for a while to really tell

no it still crashes

[glass heart]

i think i'm going to rewrite my slab allocator

[glass heart]

problem turns out not to be in the slab allocator so i'll defer it for later

where it is: in the pool header stuff

and thats why it only happens on 32-bit

[glass heart]

i think i have fixed the bug

basically i was using the HHDM on 32 bit, for the pool allocator's entries, and the HHDM implementation is faked:

• for addresses below 256 MB, it works as you'd expect
• for higher addresses, they're accessed via a 16 MB window instead that can constantly switch

instead of that I added a system to map those in completely separately

still not stable. what the fuck

this time it crashes in KiSetupRegistersThread ... bad stack allocation?!

[glass heart]

ok, i did a pretty considerable look over every HHDM offset call. other than the page cache (which I'll need to rewrite) I fixed everything

i added MmBeginUsingHHDM and MmEndUsingHHDM which actually acquire and release a lock on 32-bit (on 64-bit, they do nothing)

stable for at least 4 minutes...

i think i'm ready to move on to the next step

okay i think the firework test is stable on i386 now. it's exposing other issues (like thread starvation), but other than that, it seems stable enough. I left it running for like 8 minutes and no crashes

i'll try it again for like 30min later

[glass heart]

Well I rewrote my page cache
Hopefully it doesnt fail randomly
The amd64 version appears to have no regressions, so..

https://github.com/iProgramMC/Boron/pull/5 man this PR has more stuff than I expected...

[glass heart]

seems stable

the timer doesnt momentarily go backwards

i just have to hope it doesnt miss any timer heartbeat interrupt because that would be 4000 PIT ticks missed

*20000

[glass heart]

i have decided that one of the goals of my kernel is to expose a syscall interface that can actually simulate POSIX compliance. this means you can specify some flags (e.g. MEM_OVERRIDE) to simulate POSIX's MAP_FIXED semantics for example

[glass heart]

what the fuck

thats fucked up

status is 0 and also recursive crash

why

the answer is that it's not

but it thinks it is

just a bit later

ok, i figured out the cause

this shit actually uses a spinlock

and so you cant take page faults

WELL HERE IT IS

still need to handle some relocations, apparently i'm not handling them all

and yet i dont see any "missing relocation type" errors!

[glass heart]

i found the cause of this issue

which is that i was applying the relocations twice

once when libboron.so relocated itself on startup

and once when linking all the libraries (which includes libboron.so) and relocating them

[glass heart]

https://github.com/iProgramMC/Boron/pull/5 man this PR is huge

but im almost done

i managed to run test.exe from init.exe

[barren ocean]

lol, I assume you didn't want to include that?

[minor plinth]

What about Remnants instead of Remains?

[glass heart]

I did want to include that

[glass heart]

sure, but that message was like 2 years ago

[minor plinth]

OK

[glass heart]

i have pulled my x86 port into master

[glass heart]

i think i will need to add a pty device and ioctls for it

im not sure if i should make it part of the kernel or a separate driver

[glass heart]

the most important thing i should be pursuing though is to figure out why the process struct isnt being destructed properly

[glass heart]

was this it?

yeah seems like it was

[glass heart]

💀

so close to halloween too

also i finally got off my ass and updated limine to v10.x and flanterm

[glass heart]

i was going to do some "running boron under low memory conditions" tests on the i386 version but it turns out not to run on anything lower than 5MB due to a bug in limine: #osdev-misc-0 message

but with 4684K, the 64-bit version starts right up

(with all drivers except test.sys and all the userspace thats currently available)

this probably wont last but whatever, we take those for now

[glass heart]

im finally designing the terminal subsystem, the kernel won't implement much, just the glue logic (like the handles that programs will be writing to and reading from). i'll share a snippet of what i've come up with later

//
// The pseudo-terminal subsystem will be implemented using three
// objects:
//
// 1. The pseudoterminal object itself.
//
// It realizes the connection between the host handle (the terminal
// emulator), and the session handle(s) (the applications running
// through that terminal).  I deemed it necessary to create a whole
// new executive object type and object types because this kind of
// support logic is important to implement in the kernel.
//
// It will be an object manager object of type ``Terminal''.  It
// will not be directly writable using OSWriteFile and OSReadFile
// as it is not a file.
//
// 2. The pseudoterminal "host" file.
//
// This is a regular FCB (that file objects reference), which *can*
// be written to using OSWriteFile and OSReadFile.
//
// 3. The pseudoterminal "session" file.
//
// This is another regular FCB (that different file objects take a
// reference to), which is also writable using the OSWriteFile and
// OSReadFile system services, but it will see more traffic than
// the host side file, as several programs may be writing to it at
// once. (for example, if you run many programs with the `&` flag
// to put them in the background.
//
// Terminals can be created using the OSCreateTerminal system call.
// This will give a handle to the terminal object  (which will be
// distinct from the host and session pseudo-files).
//
// Then, handles to both host and session files can be created with
// the OSCreateIOHandlesTerminal system service, and it creates the
// handles for the host and session side (but they can pass null if 
// they don't care about one of the handles).  Both handles will be
// read/writable using the OSReadFile/OSWriteFile system service.
//```

^^^ this is my thinking of how i want to design and implement pty support in the kernel

[glass heart]

most of the basic features of the pty system have been implemented. I still need to add support for ioctls so that termios could work

and then I could get started on writing a shell making an actual proper filesystem distribution on either a tar initrd or an ext2 disk

[glass heart]

Dumping boot config.  Command line: [Root=/InitRoot Init=/InitRoot/init.exe NoInit].
        Root: /InitRoot
        Init: /InitRoot/init.exe
        NoInit: yes

i think its working how i want it

Dumping boot config.  Command line: [Root=/InitRoot Init=/InitRoot/init.exe NoInit Test1="Test 2" Test3=Test4"Test 5"Test6 Test7=Test8\ Test9].
        Root: /InitRoot
        Init: /InitRoot/init.exe
        NoInit: yes
        Test1: Test 2
        Test3: Test4Test 5Test6

and with the more complicated case

[glass heart]

I am considering renaming the boron operating system to something else. The kernel can still be known as boron, but the OS itself will be called something else

I haven't decided what it'll be called though.

[uncut cosmos]

Nanoshell2

[random sleet]

Tungsten

[glass heart]

this actually started out as a rewrite of nanoshell64 which started out as an attempt to create a new OS for 64-bit systems

[random sleet]

use an element next to Boron on the periodic table

[glass heart]

idk why this has 16 stars, its abandoned

[glass heart]

thats carbon

overused

i was thinking something more natural sounding

kinda in the theme of old ios codenames like Alpine or Snowbird

[random sleet]

aluminium

[glass heart]

those seem cool

but i wouldnt use those directly

[glass heart]

NnowTird

then I could use the Nt prefix for systemcalls instead of OS

[glass heart]

sigh

[glass heart]

ok i fixed that part. still gotta fix a bug where the mutex has too low of a level for some reason

like it crashes and says "deadlock averted" wrongly because i passed 0 to the mutex level

i have a thing where you specify the level that the mutex takes (debug only) and if you try to lock a lower level mutex while you have higher level mutexes the kernel crashes and says "deadlock averted"

NT had this too but I think they removed it

TmpBackingMemory: not locking any additional mutexes (NL)
TmpResize: if MmAllocatePool and MmFreePool use mutexes then could prove problematic, but right now I'm pretty sure they don't
TmpReadDir: NL
TmpParseDir: NL
TmpUnlink: NL
TmpMakeLink: NL

so i think its safe to assign a higher level mutex

im probably going to end up deprecating and removing this bc its kinda useless and will conflict if i end up doing like an IO stack (mounting a file as an ext2 image from an ext2 drive f ex)

instead I should add a feature to assign names to mutexes (so like this is the VAD list mutex, this is the tmpfs mutex), it'll just be a 64 bit const char*, no extra memory will be taken up

[primal gorge]

Not sure if it's useful to you, but I've found it helpful for debugging to attach names to wait entries (the thing each thread queues when blocking on something). I have a macro that fills this name with the source file and line number.

[scenic bronze]

I have a name attached to every dispatch header

and every waitable _init function takes a name parameter

[glass heart]

finished the ramdisk/tmpfs support. (writing functionality still needs to be tested, but it looks to work for now)

now i should implement a terminal emulator

[glass heart]

apparently there is a bug in libboron?? because its mapping too much of the tmpfs file

which is Strange

right. i guess BSS plays a role

[glass heart]

coming along well

need to go to the next course so will take a break but by tonight i should have a lot of things working

[glass heart]

I started work on a userspace library called CoreGraphics which operates on frame buffers of different types. This library will be the backbone of both the full screen terminal (FullScreenTerminal.exe) (and specifically the frame drawn around the terminal, because for the terminal itself I will use flanterm) as well as the eventual window manager.

[glass heart]

holy shit. literally first try

the code looks like this

(the only things i borrowed were the algorithms for line and circle drawing)

[glass heart]

ugh

now i need to draw the title text

but i'm not sure if i should implement a text rendering library or if i should just hack one together just for the full screen terminal

i think i might just write a text rendering library though

the window manager would need it too

but not sure what to call it

libCoreText?

whatever i'm gonna do it later

for now, i should setup flanterm

[glass heart]

flanterm running in userspace

bit more satisfied with this layout

the reminder to NeXTSTEP is probably intentional