OBOS (not vibecoded) | Operating System Development | Page 14

flint idol Aug 8, 2024, 10:41 AM

#

I fixed it

flint idol Aug 8, 2024, 10:56 AM

#

Theoretically, since I have all arch-specific mm functions implemented, I should be able to initialize the vmm

#

and I was able to get the allocator to work

#

I get a stack overflow into general kernel code

flint idol Aug 8, 2024, 11:19 AM

#

flint idol I get a stack overflow into general kernel code

it was an allocator bug that only happens in this very specific configuration

real pecan Aug 8, 2024, 11:23 AM

#

uacpi stress test not good enough

flint idol Aug 8, 2024, 11:23 AM

#

no I think I miscalculated the size of the kernel while initializing the kernel page tables

#

causing the mm to return a block within the kernel code

#

which should be marked as RO wth

#

if (!(phdr->p_flags & PF_R))
    ptFlags |= PT_FLAGS_READONLY;

#

oops

#

should be:

if ((phdr->p_flags & PF_R) && !(phdr->p_flags & PF_W))
    ptFlags |= PT_FLAGS_READONLY;```

#

// NOTE(oberrow): If the kernel shits it self, look here first.

#

In the pt initialization code

#

lol

real pecan Aug 8, 2024, 11:26 AM

#

lol

flint idol Aug 8, 2024, 11:39 AM

#

VMM gets halfway through init

#

then I crash while swapping out pages

#

VMM has been initialized

#

after fixing the bug in unmap

#

I just realized like half the stuff that the x86-64 port does is all just because of x86_64

#

being dumb

flint idol Aug 8, 2024, 12:21 PM

#

I need to implement timers

#

nooo

#

since I need the goldfish thing

#

but don't know the address

#

I'm going to add a limine request to the limine stub

#

to throw them at me

#

#define LIMINE_GOLDFISH_DEVICE_BUS_REQUEST { LIMINE_COMMON_MAGIC, 0x53aa5a53e788b, 0x1dbf43fc4704ea }

struct limine_goldfish_device_bus_response
{
    uint64_t revision;
    uintptr_t devbus_phys;
};

struct limine_goldfish_device_bus_request {
    uint64_t id[4];
    uint64_t revision;
    LIMINE_PTR(struct limine_goldfish_device_bus_response *) response;
};```

#

now I just need to make a response for it in the stub

devout niche Aug 8, 2024, 12:33 PM

#

Devise acpi tables for it

flint idol Aug 8, 2024, 12:34 PM

#

sorry but I don't want to become braindead (firmware dev)

#

now I need to figure out which boot info type is the goldfish device bus

#

to the qemu source I go

#

what does a control device do

#

VIRT_CTRL_MMIO_BASE

#

gotten through the tag ControlBase

#

nothing that's useful to me

#

so what I'll do is just pass the entire boot info in the request and rename it

flint idol Aug 8, 2024, 1:25 PM

#

I've implemented a very, very, basic goldfish tty thing

#

so I can now get logs without having to do a very questionable way of getting the "framebuffer"

#

I was going to implement the gold fish pic

#

so I looked at some docs for it

#

and figured that there's barely anything

flint idol Aug 8, 2024, 2:03 PM

#

wow this IRQ controller is shite

flint idol Aug 8, 2024, 2:50 PM

#

I've implemented the goldfish PIC

#

so much simpler than the LAPIC (mainly because no ACPI tables to parse)

#

but that comes at a cost

#

in that it's not a very good interrupt controller

inland radish Aug 8, 2024, 3:16 PM

#

flint idol so much simpler than the LAPIC (mainly because no ACPI tables to parse)

there's a LAPIC on the 68k?

flint idol Aug 8, 2024, 3:17 PM

#

I meant on x86-64

inland radish Aug 8, 2024, 3:17 PM

#

or is it a different thing that just happens to have the same name

#

oh but what's the goldfish PIC

flint idol Aug 8, 2024, 3:17 PM

#

no I meant that the goldfish PIC is much simpler than the x86_64 LAPIC

inland radish Aug 8, 2024, 3:17 PM

#

the legacy one?

flint idol Aug 8, 2024, 3:17 PM

#

no

inland radish Aug 8, 2024, 3:17 PM

#

oh ok

flint idol Aug 8, 2024, 3:17 PM

#

it's the one for the qemu m68k virt platform

inland radish Aug 8, 2024, 3:17 PM

#

so the 68k has the goldfish PIC

#

ah ok

flint idol Aug 8, 2024, 3:17 PM

#

well only the qemu virt platform but yeah

inland radish Aug 8, 2024, 3:18 PM

#

tbh for my thing I don't intend on porting to hardware I don't have

#

porting to virt platforms sounds boring

#

cause none of the virt specific code will be used to run real hardware

flint idol Aug 8, 2024, 3:18 PM

#

that's fair

devout niche Aug 8, 2024, 3:40 PM

#

inland radish cause none of the virt specific code will be used to run real hardware

keyronex for qemu m68k virt was a good stepping stone towards keyronex on amiga

#

that's still a nascent port mostly because of the low-quality debugger of the amiga emulator FS-UAE

inland radish Aug 8, 2024, 3:41 PM

#

my point was that you cant use the code to interface with virt devices to run real hardware

#

you'll have to write new code to interface with it

flint idol Aug 8, 2024, 3:42 PM

#

something very stupid is happening

#

I don't know what

#

basically somewhere in the kernel, something fucks up the page tables

flint idol Aug 8, 2024, 3:43 PM

#

flint idol ```c // NOTE(oberrow): If the kernel shits it self, look here first. ```

hmm

#

I was forgetting to zero the top page table on initialization

ornate ginkgo Aug 8, 2024, 8:16 PM

#

flint idol but don't know the address

The addresses are hard coded in qemu.

flint idol Aug 8, 2024, 8:16 PM

#

yeah I know

ornate ginkgo Aug 8, 2024, 8:16 PM

#

#1246702896104603698 message

flint idol Aug 8, 2024, 9:53 PM

#

I am now going to implement the gold fish rtc (unfortunately)

#

like what does

Values reported are still 64-bit nanoseconds, but they have a granularity
of 1 second, and represent host-specific values (really 'time() * 1e9')

#

even mean

#

why would the values be in nanoseconds

#

but have the granularity of a second

#

at that point just make the counters in seconds

ornate ginkgo Aug 8, 2024, 10:07 PM

#

It's for consistency with the other timer device

#

They're the same interface

flint idol Aug 8, 2024, 10:08 PM

#

how do I get a timer that runs faster than 1 second

#

or is that not supported on m68k virt

ornate ginkgo Aug 8, 2024, 10:09 PM

#

You're using the wrong device

#

There's two

flint idol Aug 8, 2024, 10:09 PM

#

how do I discover the other

ornate ginkgo Aug 8, 2024, 10:09 PM

#

One of them is your real time clock the other is your timer

flint idol Aug 8, 2024, 10:09 PM

#

how do I find the goldfish platform bus

#

or whatever it was called

ornate ginkgo Aug 8, 2024, 10:09 PM

#

You don't meme

flint idol Aug 8, 2024, 10:09 PM

#

so I assume where it is

ornate ginkgo Aug 8, 2024, 10:09 PM

#

Pretty much

#

You can get the base address of the rtc/timer devices through the boot info tags

#

But it doesn't tell you which one is which

flint idol Aug 8, 2024, 10:12 PM

#

I might be able to guess which one if I configure the timer for a fixed amount of time

#

if I set off the alarm for 1 ms, I should get it within the next ms

#

but if it's an rtc, I'd get nothing

#

since there is no alarm

ornate ginkgo Aug 8, 2024, 10:21 PM

#

It's one of those things you'll just have to hard code

flint idol Aug 8, 2024, 10:21 PM

#

oof

ornate ginkgo Aug 8, 2024, 10:21 PM

#

Although iirc the qemu virt has them both as timers?

flint idol Aug 8, 2024, 10:21 PM

#

https://github.com/qemu/qemu/blob/master/hw/rtc/goldfish_rtc.c#L71

GitHub

qemu/hw/rtc/goldfish_rtc.c at master · qemu/qemu

Official QEMU mirror. Please see https://www.qemu.org/contribute/ for how to submit changes to QEMU. Pull Requests are ignored. Please only use release tarballs from the QEMU website. - qemu/qemu

ornate ginkgo Aug 8, 2024, 10:22 PM

#

I'm not at a pc, but take a look

flint idol Aug 8, 2024, 10:22 PM

#

this says goldfish rtc

#

I'll look for goldfish timer

ornate ginkgo Aug 8, 2024, 10:23 PM

#

It's the same implementation I think

#

Take a look at the qemu virt board init

flint idol Aug 8, 2024, 10:24 PM

#

https://github.com/qemu/qemu/blob/master/hw/m68k/virt.c#L67-L70

GitHub

qemu/hw/m68k/virt.c at master · qemu/qemu

Official QEMU mirror. Please see https://www.qemu.org/contribute/ for how to submit changes to QEMU. Pull Requests are ignored. Please only use release tarballs from the QEMU website. - qemu/qemu

ornate ginkgo Aug 8, 2024, 10:29 PM

#

Looks like they're the same

flint idol Aug 8, 2024, 10:29 PM

#

I figured

ornate ginkgo Aug 8, 2024, 10:29 PM

#

Both timers and rtc, so you get to pick

flint idol Aug 8, 2024, 10:29 PM

#

so they can be either

ornate ginkgo Aug 8, 2024, 10:29 PM

#

Yes

flint idol Aug 8, 2024, 10:30 PM

#

sounds simple enough

ornate ginkgo Aug 8, 2024, 10:49 PM

#

Also remember the goldfish docs are inaccurate in places, I think the timer reg layout might be wrong

flint idol Aug 8, 2024, 10:49 PM

#

it is looking at linux kernel source

#

and at qemu source

#

wtf how is OBOS at 20k LoC

#

that's 5k more than master

#

and 4k more than the gdb stub branch

#

since I have a very minimal implementation of the gf timer

#

it should be ready now

#

whether it works or not, idk yet

#

it crashes

#

that was because the symbols that I defined were marked as weak

#

and I had forgotten to add the file that defines them

#

or maybe it's not...

#

it crashes allocating an irq object

#

thinkong

#

it crashes in the vmm here:

    page** nodes = Mm_Allocator->ZeroAllocate(Mm_Allocator, nNodes, sizeof(page*), &status);

#

that was because I forgot to implement something on the m68k

#

the irq interface was left completely untested on the m68k

#

so trying to initialize the irq object went wrong

#

fixed

flint idol Aug 8, 2024, 11:23 PM

#

currently trying to debug the reason for why the boot info passed from the limine stub is corrupted

#

but only the first two entries

#

I've tried copying out the boot info into a seperate buffer and passing that

#

(that didn't work)

#

it even appears to be corrupted in the boot loader

#

when it populates the request

#

interesting

#

it gets "corrupted" after returning from FindBootInfoTag

#

in the limine stub

#

like exactly on the rts instruction in that function

#

it gets corrupted

#

and by corrupted, I mean the first entry's base is set to 0x00100008

#

but why would that happen

#

$sp is nowhere close the first entry

#

the same thing does not happen on other calls

#

of FindBootInfoTag

#

wait it also happens before

#

wth is mofidying this

#

some funky shi be happening

main girder Aug 8, 2024, 11:53 PM

#

port to xrstation next

flint idol Aug 8, 2024, 11:53 PM

#

good idea

#

only if it has a C toolchain

#

otherwise that won't work

main girder Aug 8, 2024, 11:53 PM

#

i have to retarget a toolchain every time i want to port my thing

#

stop being a baby. just write a gcc backend and binutils support.

flint idol Aug 8, 2024, 11:54 PM

#

sorry but I'd prefer to keep my sanity

#

maybe eventually

#

though

flint idol Aug 8, 2024, 11:57 PM

#

flint idol some funky shi be happening

some funky shi indeed

#

I think I was debugging nothing this entire time

#

I was using a uint32_t as the boot info type

#

in my kernel

ornate ginkgo Aug 9, 2024, 12:20 AM

#

flint idol currently trying to debug the reason for why the boot info passed from the limin...

Yeah I was going to say you wouldn't be getting a memory map, logging or an initrd if the boot info was corrupted in the loader.

#

It would panic at the first attempt to allocate memory.

flint idol Aug 9, 2024, 1:22 AM

#

flint idol I was using a uint32_t as the boot info type

and the reason I was seeing it with the kernel's boot tag info struct instead of that of the bootloader was because I had the kernel's symbols and the bootloader's symbols loaded in gdb

#

and it preferred the kernel's struct which was formatted improperly

flint idol Aug 9, 2024, 1:55 AM

#

I hate big endian

#

I don't hate it

#

I strongly dislike

flint idol Aug 9, 2024, 1:59 AM

#

flint idol I've implemented the goldfish PIC

turns out there can be and that there are multiple

flint idol Aug 9, 2024, 2:23 AM

#

flint idol turns out there can be and that there are multiple

...but for some reason, not all are put into the boot info?

#

oh of course I have to assume the amount of PICs

#

ty to the maintainers of the qemu m68k virt platform

flint idol Aug 9, 2024, 2:54 AM

#

k this is done

#

but IRQs don't work

#

ipr is 0x20 thonk

#

sr=0x2704

#

which means the IPL is at masked

#

I was setting the IPL incorrectly

flint idol Aug 9, 2024, 3:22 AM

#

many emulation bugs I need to fix

#

(it was left completely untested)

flint idol Aug 9, 2024, 3:37 AM

#

after many bug fixes

#

it works*

#

*because of IRQL emulation memes, it just stops giving IRQs at one point

#

TODO: Fix

#

which I will do tomorrow

#

then I will make the scheduler preemptive on m68k

#

then I can implement some driver relocation code

#

and I'll be good to go

#

estimated time to finish is 2-3 days

ornate ginkgo Aug 9, 2024, 8:24 AM

#

flint idol I strongly dislike

howcome?

ornate ginkgo Aug 9, 2024, 8:24 AM

#

flint idol ...but for some reason, not all are put into the boot info?

no, you get the base address and there is always 6 of them.

#

see the ascii diagram at the top of the virt.c file in qemu

flint idol Aug 9, 2024, 11:03 AM

#

ornate ginkgo howcome?

Mistakes like using a uint32 instead of uint8 in structures are less forgiving, as you read the entirely wrong value, unlike on little endian where you would get the lowest byte

flint idol Aug 9, 2024, 11:04 AM

#

ornate ginkgo no, you get the base address and there is always 6 of them.

I ended up assuming there are six

ornate ginkgo Aug 9, 2024, 11:12 AM

#

flint idol Mistakes like using a uint32 instead of uint8 in structures are less forgiving, ...

either way thats a bug, sounds like a merit to big endian for helping you catch it sooner.

flint idol Aug 9, 2024, 11:13 AM

#

ornate ginkgo either way thats a bug, sounds like a merit to big endian for helping you catch ...

It was in the boot info tags, so it wouldn't have made it pass the m68k code anyway

flint idol Aug 9, 2024, 12:54 PM

#

flint idol \*because of IRQL emulation memes, it just stops giving IRQs at one point

it actually panicked

#

but was hanging in the text renderer

#

when I remove the framebuffer

#

it double mmu faults after a stack overflow

#

after a bit

#

of running the timer IRQ

flint idol Aug 9, 2024, 3:01 PM

#

flint idol it double mmu faults after a stack overflow

the timer irq is running way too fast

#

but I don't know why

flint idol Aug 9, 2024, 3:09 PM

#

flint idol the timer irq is running way too fast

I was reading the time wrong

flint idol Aug 9, 2024, 3:25 PM

#

well I get a bunch of timer irqs when the timer interface doesn't have any timers registered

#

otherwise, not too much (it comes to a complete halt)

flint idol Aug 9, 2024, 5:43 PM

#

flint idol otherwise, not too much (it comes to a complete halt)

it's irql emulation memes

#

idk what is exactly happening

#

ok I'm getting somewhere

#

and by that I mean two yields because of the timer, and:

Core_LowerIrqlNoThread: IRQL 1 is greater than the current IRQL, 0.```

flint idol Aug 9, 2024, 6:13 PM

#

I fixed that

#

and most IRQL memes

#

and I now preemptive multitasking

flint idol Aug 9, 2024, 6:15 PM

#

flint idol then I can implement some driver relocation code

time for this

#

then I can merge the m68k port

#

make the gdb stub faster and merge it

#

fix a bunch of merge conflicts

#

and make the vmm

#

I might slightly redo PnP after finishing the m68k port

#

so that you define the buses you want to use while discovering drivers

flint idol Aug 9, 2024, 7:00 PM

#

later on in development (a very long time) when I decide I'm bored again, I'm porting OBOS to xrstation (which includes writing a GCC backend and binutils support)

#

streaming obos

#

in voice one

#

if anyone for some reason wants to watch

flint idol Aug 9, 2024, 10:03 PM

#

I'm done the driver loader

#

except the thing is broken

#

the rela entries are corrupted

#

I suspect my linker script is the culprit

#

📎 driver_link.ld

#

@sharp pike can you check that I didn't do anything dumb in my linker script

#

please ty

sharp pike Aug 9, 2024, 10:17 PM

#

move the .dynamic section to after data

flint idol Aug 9, 2024, 10:17 PM

#

ok

sharp pike Aug 9, 2024, 10:17 PM

#

as in

#

right after the .data section

#

also discard .eh_frame* instead of just .eh_frame

flint idol Aug 9, 2024, 10:18 PM

#

ok

sharp pike Aug 9, 2024, 10:19 PM

#

and move the PT_DYNAMIC PHDR to also right after the data PHDR

#

also it is perhaps a better idea to order the pageable segments and sections also as text/rodata/data

#

instead of text/data/rodata

#

idk, other than that it lgtm

flint idol Aug 9, 2024, 10:22 PM

#

hmm ok

flint idol Aug 10, 2024, 12:42 AM

#

https://xkcd.com/1316/

xkcd: Inexplicable

ornate ginkgo Aug 10, 2024, 5:53 AM

#

flint idol the rela entries are corrupted

yeah I think there's something weird going on with m68k SOs because I get some relocations that are slightly off (within a page of where they should be).

#

I havent investigated more, I did have more success with the latest binutils (not bleeding edge lol, 2.42)

flint idol Aug 10, 2024, 10:10 AM

#

ornate ginkgo yeah I think there's something weird going on with m68k SOs because I get some r...

in my case, I'm getting garbage relocation types, which is quite weird

#

I haven't been able to figure out whether it was my problem or not yet (probably mine though)

#

I think I've found my bug

#

the values in the ehdr are being written as small endian

#

or at least some are

#

causing the relocation code to get invalid offsets to the rel(a) tables

#

because the m68k reads the values as big-endian

#

idk if this is intended behaviour with binutils, or is a bug

#

well it's rather that or I don't know how to read

#

yeah nvm

#

not the case

flint idol Aug 10, 2024, 11:02 AM

#

For future reference, never copy 64-bit elf loading code to 32-bit elf loading code, or vice versa

flint idol Aug 10, 2024, 12:09 PM

#

after fixing some of les bugs

#

it still doesn't work

flint idol Aug 10, 2024, 12:25 PM

#

it seems like the GOT is completely untouched

#

by the relocations

#

then continues to be used

#

causing the thing to crash

#

but it isn't....

#

I think I found a bug

#

case 4: // word32
    *(uint32_t*)(relocAddr) = (uint16_t)(relocResult & 0xffffffff);
    break;```

#

still crashes though...

#

in my case, it looks like it's trying to access __stack_chk_guard's entry in the GOT

flint idol Aug 10, 2024, 4:21 PM

#

flint idol in my case, it looks like it's trying to access `__stack_chk_guard`'s entry in t...

but it's trying to access entry at index zero

#

which is reserved

#

the relocation table has that variable before the GOT's base

#

fsdfdsfvgrfes

#

the GOT is 0x28 bytes ahead of the variable's supposed GOT entry

flint idol Aug 10, 2024, 9:01 PM

#

streaming obos again in voice one

#

as I try to find the bug

#

#

lol

#

from the test driver

#

it's not me

#

the linker generated a relocation

#

for the format parameter

#

which is at the GOT offset for __asan_handle_no_return

#

I don't know why the linker would act this delusional

#

let me see what x86-64's linker generates

#

it generates the same relocations, but with completely different offsets (because it's x86_64, the linker script's different and stuff)

#

currently looking at the diff between the relocations of the m68k version of the test driver vs the x86_64 version

#

all I can say so far is that the offsets of the phdrs are doubled on the m68k version (because 32-bit)

#

just noticed the align field is set to 0x2000

#

on m68k

#

which means 8k pages

#

except obos uses 4k pages

flint idol Aug 10, 2024, 9:47 PM

#

flint idol currently looking at the diff between the relocations of the m68k version of the...

so far the only significant differences were that and the fact that there are R_68K_GLOB_DAT relocs that have their offset before the GOT offset

#

which I did not find in x86_64's version

#

maybe I should try and see what clang spits out

#

nvm too much work

flint idol Aug 10, 2024, 9:53 PM

#

flint idol so far the only significant differences were that and the fact that there are `R...

I am using binutils 2.41

#

which I'm pretty sure is latest

#

2.42 is, but basically the same thing

flint idol Aug 11, 2024, 1:23 AM

#

flint idol I am using binutils 2.41

*2.42.5

#

OH MY GOD

#

nvm I was looking at the wrong ld script

#

I'll just push the code

#

merge it

#

then open up an issue (that I know I'll never close)

#

which states the issue with m68k driver loading

#

then continue to never touch the m68k port until I fix it

#

maybe that's a bad plan

#

all was going well for the m68k port before driver reloc code came in

#

imagine life but, instead of x86 taking off and becoming the main arch

#

it's the m68k

#

and we all use keyronex as our daily driver

ornate ginkgo Aug 11, 2024, 6:33 AM

#

flint idol just noticed the align field is set to 0x2000

Yeah that's part of the psabi, you can override it with linker flags though

flint idol Aug 11, 2024, 2:10 PM

#

ornate ginkgo Yeah that's part of the psabi, you can override it with linker flags though

yeah I did that

flint idol Aug 12, 2024, 12:34 AM

#

I just spent

#

ALL DAY

#

making a target for m68k-obos gcc/binutils

#

JUST FOR THE PROBLEM TO PERSIST

flint idol Aug 12, 2024, 12:34 AM

#

flint idol ALL DAY

IT WAS LITERALLY ALL DAY

flint idol Aug 12, 2024, 12:55 AM

#

at least it fails differently

#

format, as passed to OBOS_Log, is the address OBOS_Log

#

instead of __asan_handle_no_return

#

as it was with the generic m68k-elf toolchain

#

GLOB_DAT relocations are still being generated below the GOT base

#

which is really weird

#

idk if that's allowed

#

if it isn't, then that's a bug with m68k-elf-ld

#

and for some reason, it doesn't like global variables

#

the linker is retarted

#

might be the conclusion to this

empty kernel Aug 12, 2024, 1:29 AM

#

flint idol the linker is retarted

after reading through this channel, that sounds very possible

flint idol Aug 12, 2024, 1:30 AM

#

should I leave an angry bug report at binutils

#

or just let the bug thrive

#

until someone cares enough to report it

empty kernel Aug 12, 2024, 1:30 AM

#

flint idol should I leave an angry bug report at binutils

maybe

flint idol Aug 12, 2024, 1:31 AM

#

it doesn't seem to be my relocation code doing something stupid

#

as the instructions that look dumb

#

are in the (unrelocated) binary

#

after dumping them with objdump

#

but fadanoid doesn't have problems with relocations with his m68k-keyronex binutils I think

empty kernel Aug 12, 2024, 1:32 AM

#

flint idol it doesn't seem to be my relocation code doing something stupid

have you used some other relocation code to verify that?

flint idol Aug 12, 2024, 1:32 AM

#

idk where I'd find that

#

also relocation is dead simple

#

get address of relocation

empty kernel Aug 12, 2024, 1:32 AM

#

flint idol idk where I'd find that

some one elses code for reading the relocation stuff?

flint idol Aug 12, 2024, 1:32 AM

#

apply relocation usually by doing simple addition

#

and I've verified I'm reading the right values

#

as long as readelf is

#

then I am

empty kernel Aug 12, 2024, 1:33 AM

#

flint idol as long as readelf is

oh, if readelf is saying what your code is, then it is more likely a binutils bug

flint idol Aug 12, 2024, 1:34 AM

#

before I make angry bug report

#

I have one more thing

#

I'll try

flint idol Aug 12, 2024, 2:02 AM

#

angry bug report time

#

why the hell is the output binary offsetting the GOT address by 18 bytes

white mulch Aug 12, 2024, 2:17 PM

#

have you tried using clang + lld?

#

or gcc and ldd

flint idol Aug 12, 2024, 2:18 PM

#

Haven't tried clang

#

Nor lld

#

I'll try using lld to link the test driver

#

and see what it spits out

#

ld.lld-18: error: src/build/m68k/driver_link.ld:1: unknown output format name: elf32-m68k

#

so how do I get m68k support

#

nvm I don't wanna compile lldm

#

*llvm

#

as I do quite like my sanity

devout niche Aug 12, 2024, 2:27 PM

#

flint idol why the hell is the output binary offsetting the GOT address by 18 bytes

oh that sounds familiar

#

i don't remember having binutils problems

flint idol Aug 12, 2024, 2:27 PM

#

enlighten me

devout niche Aug 12, 2024, 2:27 PM

#

i do remember having problems i think even with my working compiler, until i cleaned the entire source tree and rebuilt the whole kernel

flint idol Aug 12, 2024, 2:27 PM

#

by clean the source tree, you mean just do a clean rebuild

#

like get rid of all compiled output, and recompile it

#

because I've done that several times

devout niche Aug 12, 2024, 2:28 PM

#

that's what made it spontaneously work for me

flint idol Aug 12, 2024, 2:29 PM

#

ok

#

I just verified that it's using the right compiler, and it is indeed

#

I'm pretty sure it should be using the sysv abi here

#

unlike the gcc abi or whatever it was called

#

with the generic toolchain

#

I think my problem was in ld/configure.tgt

#

nope

#

just found a swear word in m68kelf.h

#

how fun

#

currently recompiling gcc (again)

ashen goblet Aug 12, 2024, 3:03 PM

#

flint idol just found a swear word in m68kelf.h

which one is it

flint idol Aug 12, 2024, 3:04 PM

#

SVR4 m68k assembler is bitching on the `comm i,1,1' which askes for

#

gcc/config/m68k/m68kelf.h:107

#

on tag 14.2.0

#

// NOTE(oberrow): I hate myself for not adding a kmalloc or something eariler
// Goddamn is this more convinient
// The time it took to write this comment is less than the time it takes to write all the
// 'OBOS_KernelAllocator->........' shit.```

#

in obos' source

#

for reference, the line after is:

driver_header_node* node = malloc(sizeof(driver_header_node));

white mulch Aug 12, 2024, 3:08 PM

#

do you happen to have the failing code on github? I could try compiling it with clang and see whether the problem occurs there (as long as I first know how to spot it lol)

flint idol Aug 12, 2024, 3:09 PM

#

I can push it

#

I can also send the test driver source file and the ld script

#

and you can compile it with llvm

white mulch Aug 12, 2024, 3:10 PM

#

yeah that would work

flint idol Aug 12, 2024, 3:11 PM

#

then you can send the binary back and I can test it

#

📎 main.c

#

📎 driver_link.ld

#

compiled with: -ffreestanding -Wall -Wextra -fstack-protector-all -fno-builtin-memset -fvisibility=hidden -fPIC

white mulch Aug 12, 2024, 3:12 PM

#

have you changed the headers that driver uses since the last push to the m68 branch?

flint idol Aug 12, 2024, 3:12 PM

#

flint idol compiled with: `-ffreestanding -Wall -Wextra -fstack-protector-all -fno-builtin-...

linked with -nostdlib -fPIC -Wl,shared -T driver_link.ld

flint idol Aug 12, 2024, 3:12 PM

#

white mulch have you changed the headers that driver uses since the last push to the m68 bra...

I'll check the diff

#

a slight change in one of the files

#

#include <uacpi/platform/libc.h>

#

was changed to:

#

#include <uacpi_libc.h>

#

in driverId.h

#

I just pushed the code

white mulch Aug 12, 2024, 3:34 PM

#

apparently lld doesn't support m68k trolol I mean ig I can send you the object file that clang produced

flint idol Aug 12, 2024, 3:34 PM

#

sure

white mulch Aug 12, 2024, 3:34 PM

#

📎 main.o

flint idol Aug 12, 2024, 3:36 PM

#

doesn't seem like it has the same mistakes

#

it worked perfectly

#

I've narrowed the problem down to gcc doing something braindead then

#

(maybe ld is using a slightly different ABI than gcc)

#

I'm pretty sure gcc is supposed to be using the SysV ABI for the m68k

#

@white mulch what compiler options did you pass to clang

#

the same ones I sent here?

white mulch Aug 12, 2024, 3:44 PM

#

yeah those with --target=m68k-unknown-none and adding include paths + some obos defines

flint idol Aug 12, 2024, 3:44 PM

#

what defines did you add

#

I assume OBOS_DRIVER

#

or OBOS_KERNEL

white mulch Aug 12, 2024, 3:44 PM

#

and OBOS_IRQL_COUNT

flint idol Aug 12, 2024, 3:44 PM

#

oh yeah

#

it should be 8, as that's what my m68k port expects, but since you weren't compiling the kernel, it shouldn't matter

white mulch Aug 12, 2024, 3:57 PM

#

for reference the llvm command in case you want to build it at some point ```
cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON -DLLVM_TARGETS_TO_BUILD="M68k" -DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD="M68k" -DLLVM_USE_LINKER=lld -DLLVM_RUNTIME_TARGET="M68k" -DLLVM_DEFAULT_TARGET_TRIPLE="m68k-unknown-none-elf" -DLLVM_TARGET_TRIPLE="m68k-unknown-none-elf" -DLLVM_ENABLE_PROJECTS="clang;compiler-rt" -DCOMPILER_RT_INCLUDE_TESTS=OFF -DCOMPILER_RT_DEFAULT_TARGET_TRIPLE="m68-unknown-none-elf" -DCLANG_DEFAULT_RTLIB=compiler-rt -DCLANG_DEFAULT_LINKER=m68k-unknown-elf-ld -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ ../llvm

flint idol Aug 12, 2024, 3:58 PM

#

ok

#

@devout niche where is m68kv4.h, as I cannot find it in gcc's source

ashen goblet Aug 12, 2024, 4:09 PM

#

I found one here:
https://github.com/openbsd/src/blob/master/gnu/usr.bin/gcc/gcc/config/m68k/m68kv4.h

flint idol Aug 12, 2024, 4:09 PM

#

I see

#

I just realized

#

I was including m68kelf.h in m68k/obos.h

#

maybe causing m68k-obos-gcc to act like m68k-elf-gcc

#

checking for the value of EOF... configure: error: computing EOF failed

#

thinkong

#

that was in libstdc++

#

so idgaf

#

scp: open local "m68k-obos-elf-tools/m68k-obos/lib/m68000/libgcc_s.so": No such file or directory
scp: upload "m68k-obos-elf-tools/m68k-obos/lib/m68000/libgcc_s.so" to "/home/oberrow/m68k-obos-tools/m68k-obos/lib/m68000/libgcc_s.so" failed
scp: open local "m68k-obos-elf-tools/m68k-obos/lib/libgcc_s.so": No such file or directory
scp: upload "m68k-obos-elf-tools/m68k-obos/lib/libgcc_s.so" to "/home/oberrow/m68k-obos-tools/m68k-obos/lib/libgcc_s.so" failed```

#

in scp

#

idk what that's about

#

I got nothing else

#

idk why it's doing stupid stuff

#

well I'll just try running it

#

dam

#

the kernel suddenly has stack corruption

#

in read_register64

#

which has a 64-bit return value

#

which might mean I messed up something in m68k/obos.h

#

I'm sure fadanoid wouldn't mind if I copy+paste his header (m68k/keyronex.h)

#

and change one line

flint idol Aug 12, 2024, 4:48 PM

#

I fugging hate this

ashen goblet Aug 12, 2024, 4:55 PM

#

https://maps.app.goo.gl/m6QkDtUaFxof4Miq9?g_st=ic

flint idol Aug 12, 2024, 4:55 PM

#

yes I know about fugging, austria

#

I was looking at it eariler

ashen goblet Aug 12, 2024, 4:55 PM

#

What for

flint idol Aug 12, 2024, 4:55 PM

#

saw a video

#

while waiting for gcc to compile

ashen goblet Aug 12, 2024, 4:56 PM

#

oh i see

devout niche Aug 12, 2024, 4:56 PM

#

flint idol I'm sure fadanoid wouldn't mind if I copy+paste his header (`m68k/keyronex.h`)

it's fine, i did little different stealing the linux header

#

and the riscv and aarch64 headers are probably stolen from managarm, i can't remember

flint idol Aug 12, 2024, 5:05 PM

#

this doesn't make sense

#

how is there stack corruption

#

how is movel __stack_chk_guard, %fp@(-4) just doing nothing

#

I kid you not, the value at fp-4 didn't change after execution of that instruction

#

why is the stack guard zero

#

oops

#

the stack is corrupted on this line:

uint64_t half1 = (uint64_t)(read_register32(rtc, reg));

#

the stack seems to love being corrupted for no apparent reason

#

after using m68k/keyronex.h

#

which means I need to pull more stuff from keyronex's gcc patch

flint idol Aug 12, 2024, 5:43 PM

#

I tried compiling obos on the computer on which I compiled gcc, in case the binaries on my computer are out of date

#

but alas, the stack corruption persists

#

it seems as if the compiler doesn't like uint64_t

#

as whenever one is touched, the stack is corrupted

#

what if I just steal m68k/linux.h

#

and change absolutely nothing

#

(as you can see, I'm starting to get a bit desperate)

#

OH GOD

#

I accidentally ran make -j all-gcc

flint idol Aug 12, 2024, 8:26 PM

#

I think I'm getting closer to finding the bug in gcc

#

by comparing it's output with that of clang's

#

it seems to be a bug with the GOT*0 and PLT*0 relocations

#

rather in binutils, or in gcc

#

I will try to pinpoint which one has the bug

#

I wonder if compiling with light optimizations will help get rid of the bug

#

(temporarily)

flint idol Aug 12, 2024, 9:10 PM

#

case R_68K_GOT32O:
      bfd_put_32 (output_bfd, relocation, sgot->contents + got_entry_offset);
      break;```

#

these lines look sus

#

in binutils/bfd/elf32-m68k.c

#

line 3335

#

however, I do not believe that ever gets hit during the link the test driver

#

I think that when a R_68K_GOT32O relocation refers to a local symbol

#

the linker emits a R_68K_RELATIVE

#

thinkong

#

trying to understand how binutils works

#

is quite fun

#

(theoretically) I wonder how hard it would be to write my own linker

main girder Aug 12, 2024, 9:39 PM

#

its not that hard its like tying shoelaces

flint idol Aug 12, 2024, 9:39 PM

#

good to know

#

so might as well write my own, right?

#

instead of debugging binutils

#

currently streaming me debugging binutils

#

if for some ungodly reason, someone wants to watch

inland radish Aug 12, 2024, 9:42 PM

#

flint idol so might as well write my own, right?

not in that sense

#

what are you trying to do here btw?

main girder Aug 12, 2024, 9:43 PM

#

im probably the only person in this discord who has written a (static) linker so if u need advice talk me

flint idol Aug 12, 2024, 9:43 PM

#

I'm trying to find where in the world something in the toolchain messes up, giving me bs relocs

main girder Aug 12, 2024, 9:43 PM

#

but its probably inadvisable for u to do one

flint idol Aug 12, 2024, 9:43 PM

#

yeah I know

#

it was theoretical

#

this looks quite suspicious

#

could be uninitialized memory

weary hound Aug 12, 2024, 9:48 PM

#

niuuu

#

iuuu

flint idol Aug 12, 2024, 9:56 PM

#

could it be that my linker script was just shit

#

I don't think I see the same problem when linking without it

#

where did I go wrong

flint idol Aug 12, 2024, 10:13 PM

#

My linker script is indeed fucked

#

without it

#

all works fine

#

idk how it doesn't mess up clang's relocations

#

but does mess up gcc's relocations

#

but I at least don't have to debug binutils

ornate ginkgo Aug 12, 2024, 10:21 PM

#

Gcc moment

#

Interesting though, I wonder if that's the issue I had

ornate ginkgo Aug 12, 2024, 10:21 PM

#

flint idol idk how it doesn't mess up clang's relocations

You have have dynamic section and phdr in your linker script?

flint idol Aug 12, 2024, 10:21 PM

#

yes

#

PHDRS
{
    headers     PT_PHDR PHDRS;
    text        PT_LOAD FILEHDR PHDRS FLAGS((1 << 0) | (1 << 2)) ; /* Execute + Read */
    rodata      PT_LOAD FLAGS((1 << 2))            ; /* Read only */
    data        PT_LOAD FLAGS((1 << 1) | (1 << 2)) ; /* Write + Read */
    
/* 0x00010000=PF_OBOS_PAGEABLE */
    pageable_text   PT_LOAD FLAGS((1 << 0) | (1 << 2) | 0x00010000); /* Read + Execute + Pageable */
    pageable_data   PT_LOAD FLAGS((1 << 1) | (1 << 2) | 0x00010000); /* Read + Write + Pageable  */
    pageable_rodata PT_LOAD FLAGS((1 << 2) | 0x00010000);            /* Read only + Pageable */

    dynamic PT_DYNAMIC;
}```

flint idol Aug 12, 2024, 10:48 PM

#

y'know what

#

I'm sure the GNU wouldn't mind

#

if I just ~~take~~ borrow a linker script

flint idol Aug 13, 2024, 1:06 AM

#

I just crashed ld with my linker script

#

SIGSEGV

#

/DISCARD/ : {
        *(.eh_frame*)
        *(.note .note.*)
        *(.comment)
        *(.got.plt)
}```

#

after some slight modifications to my linker (copy+paste the default one)

#

it works

#

I shall now add some pageable-variants of the text/data/rodata sections

#

to the linker script

#

I think the thing that messes up the linker are the PHDRs

#

TODO: Fix that shit

#

Note: The default linker script does not use PHDRs

#

@ornate ginkgo In hopes of getting relocations to work properly for your m68k port, you could try not adding your own linker script

#

weird how my linker script worked with the clang binary

#

I guess we'll never know why

ornate ginkgo Aug 13, 2024, 1:29 AM

#

flint idol <@337185227719966721> In hopes of getting relocations to work properly for your ...

thanks I'll give that a try when I get home

flint idol Aug 13, 2024, 1:31 AM

#

as soon as I can get the default linker script to work with my pageable segments

#

I will merge the m68k port

#

and then review the gdb stub

#

merge that

#

https://github.com/OBOS-dev/obos/pull/17

GitHub

GDB Stub for x86-64 by oberrow · Pull Request #17 · OBOS-dev/obos

Implements a GDB Stub within the kernel for x86-64.

flint idol Aug 13, 2024, 1:33 AM

#

flint idol merge that

then get started on the vfs

#

after a "slight" detour

#

.

#

it took a week

flint idol Aug 13, 2024, 1:35 AM

#

flint idol then get started on the vfs

I could do that

#

or I can start another port trl

#

~~this time, to xrarch~~

#

riscv or aarch64

#

actualyl those architectures are shit

#

xtensa

#

the one used for the esp32

#

#1141057599584878645 message

#

I have talked about doing that

#

around the time of the third kernel

#

but never got around to doing it

#

completely forgot to enumerate ACPI to check for PCI

#

(TODO:)

#

this was where obos was cursed

flint idol Aug 13, 2024, 2:08 AM

#

that was a fucking lie

#

and btw, can someone enlighten me on wth a page cache does

#

I feel like I shouldn't start the VFS without having one

flint idol Aug 13, 2024, 2:10 AM

#

flint idol and btw, can someone enlighten me on wth a page cache does

or send something I can read

#

https://en.wikipedia.org/wiki/Page_cache

Page cache

In computing, a page cache, sometimes also called disk cache, is a transparent cache for the pages originating from a secondary storage device such as a hard disk drive (HDD) or a solid-state drive (SSD). The operating system keeps a page cache in otherwise unused portions of the main memory (RAM), resulting in quicker access to the contents of...

main girder Aug 13, 2024, 2:10 AM

#

it caches pages of data from a file

flint idol Aug 13, 2024, 2:10 AM

#

does file IO always go through it

main girder Aug 13, 2024, 2:11 AM

#

only cached file io

flint idol Aug 13, 2024, 2:13 AM

#

when there is a dirty page in the page cache, then an uncached file operation is done on the file it represents, what would happen

main girder Aug 13, 2024, 2:14 AM

#

usually its undefined which data ends up in the file

#

the consistency of the filesystem of course is fine though

#

it just means you want to explicitly flush the page cache for a file

#

before you do noncached io

flint idol Aug 13, 2024, 2:17 AM

#

(pseudocode):

fd1 = open("file", READ|WRITE, CACHED);
write(fd1, "whatever", 8); // page cache for the file is now dirty?
fd2 = open("file", READ|WRITE, UNCACHED); // would the kernel flush the dirty page cache here?
write(fd2, "something", 9);
close(fd2); // nothing flushed, as there is no cache
close(fd1); // dirty page cache already flushed on the open with the uncached flag set

#

or would the dirty page cache be flushed on the write to the fd that uses uncached IO

main girder Aug 13, 2024, 2:19 AM

#

idk i just have a syscall for explicit flushing

#

youd have to refer to the documentation for whatever apis youre interested in to see if they do any hidden/implicit flushes

#

i dont

#

you dont have to

#

they might for programmer convenience

flint idol Aug 13, 2024, 2:21 AM

#

I'll see what I'll do

main girder Aug 13, 2024, 2:22 AM

#

theres no reason the OS has to absolutely guarantee consistency between cached and noncached io

#

that would require crazy big locks and stuff anyway

#

if the incoherency hazard is documented then its fine

flint idol Aug 13, 2024, 2:22 AM

#

ok

main girder Aug 13, 2024, 2:23 AM

#

also remember that the IO to bring in a page of data into the cache, or to write it back to the file, is the same type of noncached io

#

so user-facing noncached io is just skipping a layer

#

and directly calling what the page cache calls to perform IO

flint idol Aug 13, 2024, 2:25 AM

#

anyway, I need to map out what I'll do in the next couple days, as I'm kind of all over the place implementing stuff

Finish the m68k port
Make the gdb stub faster (might not because lazy)
Implement mutexes, and some other locks (long overdue)
Implement a ramdisk fs driver for testing stuff
Implement page cache
Implement the VFS
Syscalls+mlibc+a bunch of other userspace stuff

flint idol Aug 13, 2024, 2:33 AM

#

flint idol anyway, I need to map out what I'll do in the next couple days, as I'm kind of a...

why not implement a couple locks now

flint idol Aug 13, 2024, 12:30 PM

#

The m68k port is done.

flint idol Aug 13, 2024, 12:47 PM

#

https://github.com/OBOS-dev/obos/pull/18

GitHub

m68k port by oberrow · Pull Request #18 · OBOS-dev/obos

Added support for the Motorola 68040 CPU, on the qemu virt platform.
This should not break compatibility with x86-64.

#

8757 additions

#

over a week and a bit

#

in exactly a week

inland radish Aug 13, 2024, 12:51 PM

#

thats cool

flint idol Aug 13, 2024, 1:04 PM

#

I just merged the gdb stub and m68k port

#

but now I need to fix master

#

as the gdbstub branch moved scheduler/dpc.h to irq/dpc.h

#

but the m68k port was based on master

cerulean cipher Aug 13, 2024, 1:05 PM

#

flint idol 8757 additions

impressive!

flint idol Aug 13, 2024, 1:06 PM

#

ty

#

(tbf though, a lot of the additions were just me copy+pasting from my x86-64 port)

#

The gdb stub no longer works

#

it crashes the kernel

#

something (com driver?) registered an irq object and forgot to add a callback

#

irq_obj->handler was nullptr

#

if (irq_obj->handler)
{
    irq_obj->handler(
        irq_obj,
        frame,
        irq_obj->handlerUserdata,
        oldIrql2);
}```

#

............

#

Page fault at 0x0000000000000000 in kernel-mode while to execute page at 0x0000000000000000, which is unpresent.

#

it's on line 80 though

#

nvm that's just the stack trace lying

#

I've spent too much time in m68k

#

64-bit pointers look weird

#

relocations seem to be broken on x86-64 now

flint idol Aug 13, 2024, 1:36 PM

#

flint idol relocations seem to be broken on x86-64 now

nvm, making a symbol weak with the visibility default breaks stuff

#

suddenly the gdb stub's thread view is broken

#

this is weird

#

I could've sworn I implement detaching

#

I had forgotten to add it to the packet map

flint idol Aug 13, 2024, 1:48 PM

#

flint idol suddenly the gdb stub's thread view is broken

it works now

flint idol Aug 13, 2024, 2:04 PM

#

time to implement mutexes

flint idol Aug 13, 2024, 4:48 PM

#

after a slight detour

#

I will now implement mutexes

devout niche Aug 13, 2024, 5:01 PM

#

flint idol https://github.com/OBOS-dev/obos/pull/18

Well done

flint idol Aug 13, 2024, 5:01 PM

#

ty

flint idol Aug 13, 2024, 5:27 PM

#

is this mutex implementation good:
To lock:

Spin on the lock for a bit, if it succeeded in spinning, return success
Otherwise, add the thread to a list of waiting threads in the mutex, and block.
Return success
To try locking:
If the lock is already acquired, abort.
Otherwise, lock as normal.
On release:
If this thread was the one who locked the mutex before, continue. Otherwise, fail
Release the locked variable
Pop a thread from the waiting list, and unblock it.
Return success.

#

https://github.com/OBOS-dev/obos/tree/locks

#

can someone review my mutex code

#

https://github.com/OBOS-dev/obos/blob/locks/src/oboskrnl/locks/mutex.c

#

time to implement semaphores

#

in what case would a semaphore be used

#

anyway

#

it's pretty simple though

flint idol Aug 13, 2024, 6:25 PM

#

I have implemented semaphores

#

hopefully correctly

#

and if anyone could advise me on how to test mutexes and semaphores, that would be useful

inland radish Aug 13, 2024, 6:28 PM

#

flint idol and if anyone could advise me on how to test mutexes and semaphores, that would ...

spawn three threads

#

initialize the semaphore

#

release it three times in a row

#

in the main thread

#

and each thread should go

#

your count should be 1 and the initial state is 3

#

thats how i think you can test semaphores

flint idol Aug 13, 2024, 6:28 PM

#

ok

flint idol Aug 13, 2024, 6:52 PM

#

@real pecan your thing is broken

#

this time it's the bootloader

#

cmdline="--enable-kdbg \"--root=ramdisk\""

#

in the config

#

causes it to hang

#

at boot

real pecan Aug 13, 2024, 6:56 PM

#

ill take a look but u can just do '--enable-kdbg "--root=ramdisk"'

flint idol Aug 13, 2024, 6:56 PM

#

when booting on UEFI it gives an error

flint idol Aug 13, 2024, 6:56 PM

#

real pecan ill take a look but u can just do `'--enable-kdbg "--root=ramdisk"'`

ok

#

real pecan Aug 13, 2024, 6:57 PM

#

yeah because it doesnt support escaping

#

why it would hang on bios tho i have no idea

sharp pike Aug 13, 2024, 7:08 PM

#

this is why i never bothered with quotations in the Limine config

#

just error prone and useless

real pecan Aug 13, 2024, 7:11 PM

#

they arent mandatory if u dont use spaces

#

but yeah its kinda annoying to parse

flint idol Aug 13, 2024, 7:36 PM

#

I made an argument parser for the kernel

#

Current arguments are:

--enable-kdbg: Enables the kernel debugger at boot
--help: Displays this help message.```

real pecan Aug 13, 2024, 8:20 PM

#

does enable-kdbg take in a separate argument for some reason?

flint idol Aug 13, 2024, 8:43 PM

#

real pecan does enable-kdbg take in a separate argument for some reason?

No

#

(t yet)

real pecan Aug 13, 2024, 10:08 PM

#

ah

flint idol Aug 14, 2024, 1:02 AM

#

my semaphores don't work

#

Kernel Panic on CPU 1 in thread 7, owned by process -1. Reason: OBOS_PANIC_ASSERTION_FAILED. Information on the crash is below:
Assertion failed in function CoreH_ThreadListRemove. File: /home/oberrow/Code/obos/src/oboskrnl/scheduler/thread.c, line 165. cur

#

unfortunately

#

maybe a bug with the remove/append function

#

the linker list looks very goofy

#

dam

#

it looks like heap corruption of some kind

#

kasan is on

#

but the semaphore is on the stack-

#

oh yeah

#

is the irql supposed to be raised while a semaphore is raised

#

idk much, but I do know something is fucking up the linked list

#

that contains threads waiting on the semaphore

#

I "fixed" it by moving where the thread is removed from the queue

#

#

semaphores seem to work fine

flint idol Aug 14, 2024, 1:21 AM

#

flint idol anyway, I need to map out what I'll do in the next couple days, as I'm kind of a...

next up is:

Implement a ramdisk driver
Page cache + VFS

#

(after I test mutexes)

ornate ginkgo Aug 14, 2024, 6:19 AM

#

nice, congrats on the m68k support

#

why is your pmm and elf loader arch specific?

flint idol Aug 14, 2024, 9:58 AM

#

ornate ginkgo why is your pmm and elf loader arch specific?

Too much work to make them arch independent

#

I might on the next time I port obos

#

Or when ever I have the time

main girder Aug 14, 2024, 9:59 AM

#

flint idol Too much work to make them arch independent

not really!

ornate ginkgo Aug 14, 2024, 10:00 AM

#

the elf loader should be easy, I cant think of any reason a pmm would need any arch specific hooks at all

main girder Aug 14, 2024, 10:00 AM

#

ornate ginkgo the elf loader should be easy, I cant think of any reason a pmm would need any a...

i can

ornate ginkgo Aug 14, 2024, 10:00 AM

#

go on then

main girder Aug 14, 2024, 10:01 AM

#

#1026090868303732766 message

#

this code uses some arch-defined macros while dereferencing the containing page for a tracking entry for a reclaimed page

#

upon allocation

#

thats physical memory allocation so u could say its pmm.

ornate ginkgo Aug 14, 2024, 10:04 AM

#

look I'm sure there's a point to be made there, but I dont know whats going on in that code

main girder Aug 14, 2024, 10:05 AM

#

allocate page frame -> none are on free list, have to reclaim one -> ends up there

flint idol Aug 14, 2024, 3:16 PM

#

Whenever I say too much work, I always mean I'm too lazy to do it

flint idol Aug 14, 2024, 4:19 PM

#

I have written mutex tests

#

and fixed a couple bugs with them

#

any other locks that are worth implementing

#

before I go on to vfs stuff

inland radish Aug 14, 2024, 4:31 PM

#

flint idol any other locks that are worth implementing

perhaps events (synchronization and notification)

#

Programming interfaces that support the event object include:

KeInitializeEvent - Initialize an event object
KePulseEvent - Set/reset event object state atomically
KeReadStateEvent - Read state of event object
KeResetEvent - Set event object to Not-Signaled state
KeSetEvent - Set event object to Signaled state

flint idol Aug 14, 2024, 4:40 PM

#

ok

#

typedef struct event {
    atomic_flag signaled;
    thread_list waiting;
    event_type type;
} event;```

#

I think that's all I'll need for the event object

#

maybe I should change the atomic flag to an atomic bool

flint idol Aug 14, 2024, 5:02 PM

#

obos_status Core_EventPulse(event* event, bool boostWaitingThreadPriority);
bool        Core_EventGetState(const event* event);
obos_status Core_EventReset(event* event);
obos_status Core_EventSet(event* event, bool boostWaitingThreadPriority);
obos_status Core_EventClear(event* event);
obos_status Core_EventWait(event* event);```

#

I think that's all the API I'll need for events

flint idol Aug 14, 2024, 5:47 PM

#

Events seem to work

#

at notification-based ones do

#

I'll see sync type events

#

both event types work

#

which means I can merge my PR

inland radish Aug 14, 2024, 5:56 PM

#

@flint idol i suggest you have just one wait instead of many (with a dispatch header or whatever) and a multi-wait for any object (and multi-wait for all objects if you want)

flint idol Aug 14, 2024, 5:56 PM

#

good idea

#

obos_status Core_WaitOnObject(struct waitable_header* obj);
obos_status Core_WaitOnObjects(size_t nObjects, ...);
obos_status Core_WaitOnObjectsPtr(size_t nObjects, struct waitable_header* objs);
obos_status CoreH_WakeWaitingThreads(struct waitable_header* obj, bool all);```

#

should be good

#

I renamed wake waiting threads to signal waiting threads

#

and added a parameter which specifies whether to boost the priority of the thread(s) that are signaled

#

the basic waiting API works

#

However, I need to implement wait on objects

#

well I have a list of threads

#

struct waitable_header
{
    thread_list waiting;
};

#

in waitable_header

#

and if I have multiple objects that'd be kind of complicated

#

I'll add a field to struct thread

#

*two fields

#

how many objects the thread is waiting on

#

and the amount of objects that have signaled the thread

#

CoreH_SignalWaitingThreads will increment that count

flint idol Aug 14, 2024, 6:31 PM

#

flint idol how many objects the thread is waiting on

and if it sees that it is equal to this count, the thread gets unblocked

#

otherwise it doesn't

flint idol Aug 14, 2024, 7:42 PM

#

wait for objects works

#

nvm

#

as it never waited in the first place

#

use after free thonk

#

oops

#

#

now time to remove all the verbose logs

#

I just merged the locks branch

#

https://github.com/OBOS-dev/obos/pull/19/

GitHub

Add new locks by oberrow · Pull Request #19 · OBOS-dev/obos

Adds mutexes and semaphores.

flint idol Aug 14, 2024, 9:48 PM

#

time for the vfs

flint idol Aug 14, 2024, 11:05 PM

#

I think I'll use USTAR as the format for my initrd

#

because it's awesome

flint idol Aug 15, 2024, 2:30 AM

#

I've added a way to specify the path of additional drivers to load through the command line

#

rn it's only the names of modules

#

but once I get the initrd interface done, it will also be relative paths from the initrd root of drivers to load

#

this way, if one wanted, they could boot OBOS without an initrd

#

However, these modules are unconditionally loaded

flint idol Aug 15, 2024, 2:33 AM

#

flint idol but once I get the initrd interface done, it will also be relative paths from th...

by initrd interface, I mean the functions in the function table

#

I am done the hard part of the initrd

#

which wasn't quite hard at all

#

so all that's left are the interface functions

#

which there are only 14 of

#

some of which don't need to be implemented

#

as the initrd is a RO filesystem

#

and some functions are for writing

#

sometime while I'm making the vfs, I'm adding a command line parameter for the UUID of the partition to mount as root

#

and after the vfs, I'm adding a command line parameter for the init program to load and run

#

then I'll go on to implement a lot of syscalls

#

then port mlibc

#

then suffer

#

estimated time to finish: I have no fugging idea

main girder Aug 15, 2024, 2:40 AM

#

gl

flint idol Aug 15, 2024, 2:40 AM

#

ty

empty kernel Aug 15, 2024, 4:12 AM

#

flint idol because it's awesome

Yeah, its really simple

flint idol Aug 15, 2024, 10:30 AM

#

decided to test obos on real hw

#

and I got stack corruption

#

at least it's in the same area each time though

#

0xffffffff80034930: Core_Yield at /home/oberrow/Code/obos/src/oboskrnl/scheduler/schedule.c:298

#

in the BSP idle thread

flint idol Aug 15, 2024, 10:51 AM

#

uStressTester strikes once again

flint idol Aug 15, 2024, 11:16 AM

#

As usual, -fanalyzer catches basically nothing

empty kernel Aug 15, 2024, 11:19 AM

#

flint idol As usual, `-fanalyzer` catches basically nothing

What's -fanalyzer?

flint idol Aug 15, 2024, 11:19 AM

#

GCC's static analyzer

empty kernel Aug 15, 2024, 11:21 AM

#

Ah so that would explain all the various analyzer* files that take ages to build every time I build GCC.

flint idol Aug 15, 2024, 11:21 AM

#

I think you can disable it

empty kernel Aug 15, 2024, 11:23 AM

#

Nah. Too much effort. Anyway, I might use it some day and I really wouldn't be happy if I had to rebuild GCC for it.

flint idol Aug 15, 2024, 11:24 AM

#

nvm, you can't disable it at gcc compile time

#

my analyzer output is 3000 lines

#

I commented out a line, now the stack is corrupted somewhere else

#

nvm the address is just different

#

the function in which it crashes in is the same

flint idol Aug 15, 2024, 12:38 PM

#

flint idol because it's awesome

but very simple

flint idol Aug 15, 2024, 12:54 PM

#

flint idol but very simple

.. a bit too simple

lean glen Aug 15, 2024, 12:54 PM

#

Tar works fine ime

flint idol Aug 15, 2024, 1:35 PM

#

#

Directory traversal in the initrd drivers seems to work

#

in this case, it's testing the list_dir callback

flint idol Aug 15, 2024, 2:02 PM

#

initrd driver is done

#

time for the VFS

#

no way this is that complicated

#

I have mount point

#

mount point has tree of vnodes

#

vnode has pointer to inode and a filename+path

#

inode has file data, perms, and type

#

inode probably also has page cache things

#

cached reads go through the page cache, and if there is no page cache hit, ask the fs driver for the data, then add that to the page cache

main girder Aug 15, 2024, 2:07 PM

#

flint idol vnode has pointer to inode and a filename+path

:)

flint idol Aug 15, 2024, 2:07 PM

#

(right?)

main girder Aug 15, 2024, 2:07 PM

#

pointer to inode?

#

are your vnodes a 1:1 structure where a vnode in memory corresponds in a one-to-one fashion with a file on disk

flint idol Aug 15, 2024, 2:08 PM

#

no

weary hound Aug 15, 2024, 2:08 PM

#

vnodes don't contain their names or paths

#

because a single vnode might be reachable from multiple names/paths

#

via hard links

flint idol Aug 15, 2024, 2:09 PM

#

then I got my terminology wrong

main girder Aug 15, 2024, 2:09 PM

#

this is still fine if what hes calling "inode" is what normal people call a vnode and what hes calling "vnode" is what normal people call a dirent cache entry

weary hound Aug 15, 2024, 2:09 PM

#

if thats the case then yeah

main girder Aug 15, 2024, 2:09 PM

#

flint idol no

which i suspect based on this

flint idol Aug 15, 2024, 2:10 PM

#

I got my terminology wrong

flint idol Aug 15, 2024, 2:11 PM

#

flint idol cached reads go through the page cache, and if there is no page cache hit, ask t...

cached writes go through the page cache, and if there's no page cache hit, make an uncached write via the fs driver, then fill the page cache

#

uncached io just goes through the fs driver

#

after some reading on vnodes and dirents it makes sense

flint idol Aug 15, 2024, 7:01 PM

#

just pushed all of my structs

#

for the VFS

#

(maybe not all, but most I know I'll use)

#

I also implemented a function to lookup a dirent

#

with a path

#

hopefully I implemented it correctly

#

a dirent is basically just a tree node

#

I also implemented some more string manipulation routines

#

for allocating/freeing/append to/resizing/comparing strings

#

https://github.com/OBOS-dev/obos/blob/vfs/src/oboskrnl/utils/string.h

flint idol Aug 15, 2024, 7:08 PM

#

flint idol hopefully I implemented it correctly

https://github.com/OBOS-dev/obos/blob/vfs/src/oboskrnl/vfs/dirent.c#L24

#

do note that I haven't modified it to query any sort of name cache

#

that's only like a couple lines to do though

#

struct mount contains the name cache for the current mount point

#

so if I hit a mount point (that's a part of the directory) while traversing the tree, I just do an RB lookup within the name cache

#

and if there's a hit, I can return early

#

a namecache entry will contain a list of directory entries that it refers to, as well as the vnode it refers to

ashen goblet Aug 15, 2024, 7:14 PM

#

Sorry for interrupting but i recommend having a look at what keyronex did for unmounting (i used it for my implementation aswell because it’s quite the clever solution)

flint idol Aug 15, 2024, 7:20 PM

#

ashen goblet Sorry for interrupting but i recommend having a look at what keyronex did for un...

ok

#

also it was np, as I was done rambling on my progress anyway

ashen goblet Aug 15, 2024, 7:21 PM

#

nice

#

you‘re making great progress

flint idol Aug 15, 2024, 7:22 PM

#

indeed

ashen goblet Aug 15, 2024, 7:22 PM

#

👏

flint idol Aug 15, 2024, 7:22 PM

#

unfortunately, this will slow down tremendously after 2-3 weeks, as school is starting nooo

ashen goblet Aug 15, 2024, 7:22 PM

#

I feel you there, same for me

#

Its already bad on my end because i‘m doing an internship

flint idol Aug 15, 2024, 7:23 PM

#

but is it just me, or do more people start osdev during the (northern hemisphere) summer

#

for example I did

#

during the summer

#

and a bunch of people have joined this server this summer

ashen goblet Aug 15, 2024, 7:24 PM

#

more free time, more time for stupid ideas - so probably?

flint idol Aug 15, 2024, 7:26 PM

#

that's what I think

#

anyway

flint idol Aug 15, 2024, 8:13 PM

#

flint idol a namecache entry will contain a list of directory entries that it refers to, as...

nvm, it needs not be a list

empty kernel Aug 15, 2024, 8:17 PM

#

flint idol but is it just me, or do more people start osdev during the (northern hemisphere...

Even though I'm in the southern hemisphere, I also started at that time of year. I have no clue why though. June/July is normally when I have less time.

flint idol Aug 15, 2024, 8:17 PM

#

interesting

#

static namecache_ent* namecache_lookup_internal(namecache* nc, const char* path)
{
    namecache_ent what = { .path=(char*)path };
    return RB_FIND(namecache, nc, &what);
}
static dirent* namecache_lookup(namecache* nc, const char* path)
{
    namecache_ent* nc_ent = namecache_lookup_internal(nc, path);
    if (!nc_ent)
        return nullptr;
    return nc_ent->ent;
}```
Namecache lookup functions are done

#

and for some reason clangd is dead

#

I've made it so that VfsH_DirentLookup inserts the dirent in the namecache after finding it

#

time to implement mount

#

then test the two functions

#

or I could just add some nodes to the root node

#

then test VfsH_DirentLookup

flint idol Aug 15, 2024, 9:03 PM

#

DirentLookup works, at least on one level:

dirent* name1 = VfsH_DirentLookup("/name1.txt");
dirent* name2 = VfsH_DirentLookup("/name2.txt");
if (name1)
    OBOS_Debug("Found dirent at /%s.\n", OBOS_GetStringCPtr(&name1->name));
if (name2)
    OBOS_Debug("Found dirent at /%s.\n", OBOS_GetStringCPtr(&name2->name));```

#

but not on two

#

I fixed it on two levels

#

time to test on three levels

#

then if it works on three levels, it works on any amount of levels

flint idol Aug 15, 2024, 9:28 PM

#

so it wasn't working on three levels

#

then I fixed that

#

then it stopped working with three levels xD

#

fixed that

#

time to test with four levels

#

fixed*
*I need to fix my tokenization

#

and multiple children doesn't work

#

for some reason it traverses one node down the tree

#

fixed

flint idol Aug 15, 2024, 10:20 PM

#

time to implement mounting

#

(this'll be annoying)

#

I also got a new pfp

elder hornet Aug 15, 2024, 10:24 PM

#

flint idol I also got a new pfp

Nice one :D

flint idol Aug 15, 2024, 10:24 PM

#

ty

elder hornet Aug 15, 2024, 10:29 PM

#

flint idol time to implement mounting

I haven't done a VFS yet, what does this involve?

flint idol Aug 15, 2024, 10:29 PM

#

not much

#

just iterating over a filesystem's directory entries

#

and slapping them in a mountpoint

elder hornet Aug 15, 2024, 10:30 PM

#

ah okay. Seems simple enough

flint idol Aug 15, 2024, 10:30 PM

#

famous last words

flint idol Aug 15, 2024, 11:51 PM

#

after locking in

#

it surprisingly wasn't that hard

#

(because I already did the hard work in VfsH_DirentLookup)

#

I'm almost done, I just need to add code to resolve symbolic links

flint idol Aug 16, 2024, 12:18 AM

#

flint idol I'm almost done, I just need to add code to resolve symbolic links

that's done

#

which means mounting is implemented

#

all I gotta do is test it

flint idol Aug 16, 2024, 1:16 AM

#

#

I needed to fix some bugs, but otherwise it works perfectly

#

(based on the logs)

#

it does not work perfectly.

flint idol Aug 16, 2024, 1:53 AM

#

I fixed that bug

#

#

Vfs_Mount("/", nullptr, &initrd_dev, nullptr);
const char* const pathspec = "/test_folder/subdir/other_file.txt";
dirent* found = VfsH_DirentLookup(pathspec);
if (found)
    OBOS_Debug("Found %s.\n", pathspec);```

#

Tomorrow I will work on file descriptors

#

and file IO

#

and also device IO through special files

#

(not that the latter is special or anything)

#

My thread has the most messages of any progress report thread

#

me when starting a new rewrite:

#

(no, I'm not rewriting)

#

I don't think I'm making any more rewrites of OBOS

chilly osprey Aug 16, 2024, 2:14 AM

#

ive told myself that many times

flint idol Aug 16, 2024, 2:14 AM

#

...

#

how many rewrites have you had

chilly osprey Aug 16, 2024, 2:14 AM

#

the rewrite cycle never stops the time between rewrites just grows

chilly osprey Aug 16, 2024, 2:14 AM

#

flint idol how many rewrites have you had

are we counting like

#

hello world kernels when i was 14

#

that did nothing

flint idol Aug 16, 2024, 2:14 AM

#

ones that did something

chilly osprey Aug 16, 2024, 2:14 AM

#

4 maybe

#

im not sure

flint idol Aug 16, 2024, 2:15 AM

#

OBOS still has more (that actually did something)

#

half of which were unstable and would crash often

chilly osprey Aug 16, 2024, 2:18 AM

#

usually i would work really hard for a few weeks

#

then everything would fall apart

#

and i lost all my motivation

#

then a few months later i'd come back and rewrite

flint idol Aug 16, 2024, 1:14 PM

#

flint idol Tomorrow I will work on file descriptors

time to start on this

flint idol Aug 16, 2024, 1:42 PM

#

before I do that, I need to fix bugs with the namecache

flint idol Aug 16, 2024, 2:11 PM

#

time to implement uid and gid

#

done

#

(it's a typedef)

#

typedef uint32_t uid, gid;
#define ROOT_UID 0
#define ROOT_GID 0```

flint idol Aug 16, 2024, 2:38 PM

#

just fixed a bug with mounting

#

specifically resolving symlinks

#

if a symlink pointed to another symlink, but that other symlink hadn't been resolved yet, the original symlink's vnode would be nullptr

#

by resolve, I mean 'get the vnode it points to'

#

obos_status  Vfs_FdOpen(fd* const desc, const char* path, uint32_t oflags);
obos_status Vfs_FdWrite(fd desc, const void* buf, size_t nBytes, size_t* nWritten);
obos_status  Vfs_FdRead(fd desc, void* buf, size_t nBytes, size_t* nRead);
obos_status  Vfs_FdSeek(fd desc, off_t off, whence_t whence);
uoff_t    Vfs_FdTellOff(fd desc);
// Returns OBOS_STATUS_EOF on EOF, OBOS_STATUS_SUCCESS if not on EOF. anything else is an error.
obos_status   Vfs_FdEOF(fd desc); 
obos_status Vfs_FdFlush(fd desc);
obos_status Vfs_FdClose(fd desc);```
This should be enough API for file descriptors, apart from the async io functions that I'll make in a bit.

#

I'll also throw in ioctl

flint idol Aug 16, 2024, 3:00 PM

#

Open was fairly simple

#

lookup the dirent, set desc->vnode accordingly, then set some fd flags according to the perms and the current uid and gid

flint idol Aug 16, 2024, 3:27 PM

#

read is also farily trivial

#

if desc->flags & UNCACHED, do uncached read

#

otherwise do cached read

#

a cached read queries the page cache

#

for chunks of the data to read

#

it fills in the chunks that were found in the page cache

#

as for those that were not found

#

an uncached read is done

#

then they're filled

#

at least this is what is done for files

flint idol Aug 16, 2024, 6:31 PM

#

cached reads seem to work

#

const char* const pathspec = "/test_folder/file.txt";
fd file = {};
Vfs_FdOpen(&file, pathspec, FD_OFLAGS_READ_ONLY);
char buf[16];
Vfs_FdRead(&file, buf, 6, nullptr);
Vfs_FdSeek(&file, 0, SEEK_SET);
Vfs_FdRead(&file, buf, 13, nullptr);
OBOS_Debug("%s:\n", pathspec);
printf("%s\n", buf);

flint idol Aug 16, 2024, 6:44 PM

#

flint idol ```c const char* const pathspec = "/test_folder/file.txt"; fd file = {}; Vfs_FdO...

(if anyone was wondering why I had the first read, it was test to see if the page cache would be consulted on the second read)

lean glen Aug 16, 2024, 8:22 PM

#

virgin pagecache vs chad everything is a tmpfs

flint idol Aug 16, 2024, 8:41 PM

#

time to implement writing

#

basically the same as reading

#

except the opposite

lean glen Aug 16, 2024, 8:42 PM

#

writing is likely gonna be the trickiest

#

you need to ensure everything is synced properly

flint idol Aug 16, 2024, 8:44 PM

#

my page cache is shit

#

it's a linked list

#

of:

typedef struct pagecache_ent
{
    char* data;
    size_t sz;
    size_t fileoff;
    bool dirty;
    LIST_NODE(pagecache, struct pagecache_ent) node;
} pagecache_ent;```

#

except nothing is done to sort the entries

#

or speed up search

#

or even avoid overlapping regions

#

which could break writing

lean glen Aug 16, 2024, 8:45 PM

#

wdym search

#

wouldnt it be a good idea to have a pointer to a cache entry in your dent

flint idol Aug 16, 2024, 8:47 PM

#

it is, except the page cache doesn't contain the entire file

#

I'll just unshittify the page cache

lean glen Aug 16, 2024, 8:50 PM

#

flint idol it is, except the page cache doesn't contain the entire file

yeah but why are you searching

flint idol Aug 16, 2024, 8:51 PM

#

because a page cache entry stored a region within a file

#

not the entire thing

lean glen Aug 16, 2024, 8:51 PM

#

oh right

#

I was thinking of a case where all accesses are from offset 0, then 0x1000, then 0x2000, etc.

#

so you could just merge adjacent page cache entries

flint idol Aug 16, 2024, 8:52 PM

#

however I am changing it to just be from offset zero up to the offset of the biggest access made on the file

#

and it'll freed be when there are no longer file descriptors refering to it

lean glen Aug 16, 2024, 8:54 PM

#

yeah keep a refcnt

flint idol Aug 16, 2024, 8:54 PM

#

typedef struct pagecache
{
    char* data;
    size_t sz;
    dirty_pc_list dirty_regions;
    size_t refcnt;
} pagecache;```

#

this should be good

lean glen Aug 16, 2024, 8:54 PM

#

or just keep a refcnt in your vnode

#OBOS (not vibecoded)