OBOS (not vibecoded) | Operating System Development | Page 16

flint idol Aug 22, 2024, 6:45 PM

#

because yes

real pecan Aug 22, 2024, 6:45 PM

#

Its the only right one

flint idol Aug 22, 2024, 6:45 PM

#

it's not like it's doo doo water or anything

#

because it's not

#

it's only one of the best of the worst

#

uDriverInterface

real pecan Aug 22, 2024, 6:49 PM

#

u mean oDriverInterface

flint idol Aug 22, 2024, 6:49 PM

#

sure

#

it might be fun to make that sometime

#

except it won't

#

but it might

short mortar Aug 22, 2024, 6:50 PM

#

There needs to be some userspace driver interface

#

(Because that's how my drivers work)

flint idol Aug 22, 2024, 6:52 PM

#

@real pecan you said this:

3. If staggered spin up is supported set the SUD bit in sata status register.
4. Wait for about 1ms for DET in sata status to be set to 3.```
does that mean to wait for SSTS.DET to become three only if staggered spin up is supported

#

or do I always do it

short mortar Aug 22, 2024, 6:53 PM

#

If you've just reset the port/controller, I think it's always better to wait (?)

flint idol Aug 22, 2024, 6:53 PM

#

I'll just wait

#

just in case

real pecan Aug 22, 2024, 6:53 PM

#

flint idol <@353593948523986945> you said this: ``` 3. If staggered spin up is supported se...

no it doesnt

flint idol Aug 22, 2024, 6:54 PM

#

ok

#

HBA_PORT* hPort = HBA->ports + port;
hPort->cmd |= BIT(4); // set FRE
if (HBA->cap.sss)
    hPort->cmd |= BIT(1); // set SUD
timer_tick deadline = CoreS_GetTimerTick() + CoreH_TimeFrameToTick(1000);
while ((hPort->ssts & 0xf) != 0x3 && deadline < CoreS_GetTimerTick())
    OBOSS_SpinlockHint();```

#

this is what I have so far for port init

short mortar Aug 22, 2024, 6:57 PM

#

I think you should also wait for the port to not be busy (?)

flint idol Aug 22, 2024, 6:57 PM

#

I wasn't done

#

HBA_PORT* hPort = HBA->ports + port;
hPort->cmd |= BIT(4);
if (HBA->cap.sss)
    hPort->cmd |= BIT(1);
timer_tick deadline = CoreS_GetTimerTick() + CoreH_TimeFrameToTick(1000);
while ((hPort->ssts & 0xf) != 0x3 && deadline < CoreS_GetTimerTick())
    OBOSS_SpinlockHint();
if ((hPort->ssts & 0xf) != 0x3)
    continue;
hPort->serr = 0xffffffff;
while (hPort->tfd & 0x88)
    OBOSS_SpinlockHint();
OBOS_Debug("Done port init for port %d.\n", port);```

#

port init works on virtualbox and qemu

#

I will now try to enable IRQs and see what happens when I send a command

#

I just need to set the thing bases

#

command list and fis bases

flint idol Aug 22, 2024, 8:08 PM

#

I got identify ata working on qemu

#

But I still cannot receive irqs on real hw

#

I don't think it's the IRQs

#

since I just saw that I was busy looping

#

the command never completes

#

since my third kernel's ahci driver does work on this device

#

I will compare what this driver and the other do differently

flint idol Aug 22, 2024, 9:45 PM

#

I hate AHCI

#

I am no closer than I was to solving this problem than I was one day

#

Enable AHCI mode by setting GHC.AE.

Perform BIOS handoff if supported.

Make sure all ports are idle (ST, CR and FR, FE are cleared)

Perform a standard AHCI reset by setting the GHC.HR bit and wait for it to become 0.

Enable AHCI mode by setting GHC.AE.
For each implemented port:

Set both command list & fis receive to a valid physical address.

Set the fis receive (FE) bit in the port register (otherwise PxTFD.STS.BSY will be set to 1 forever).

If staggered spin up is supported set the SUD bit in sata status register.

Wait for about 1ms for DET in sata status to be set to 3.

If it's not set within the timeframe, port has no device attached, so we continue to the next port.

Clear the port error register to 0xFFFFFFFF (otherwise again it will be stuck in BSY forever).

Spin on PxTFD.STS.DRQ and PxTFD.STS.BSY, they must be cleared within a small amount of time (after the device has finished transferring the initial FIS with signature and stuff)

#

Checklist for HBA:
1 - done
2 - done
3 - done
4 - done
5 - done

#

Checklist for ports:
1 - done
2 - done
3 - done
4 - done
5 - done
6 - done
7 - done

#

I then set is to 0xffffffff and ie to 0xffffffff

#

then after that loop HBA.ghc.ie is set to true

#

then ATA_IDENTIFY is issued and it hangs

#

because the command never completes

white mulch Aug 22, 2024, 9:56 PM

#

flint idol I hate AHCI

I haven't yet done ahci myself so I don't hate it yet KEKW I have some other things I want to try doing first like ufs because it seems pretty nice and then you just use some scsi commands with it to read the LU's

flint idol Aug 22, 2024, 9:57 PM

#

yet

#

is the keyword

#

what if I just copy+paste some code

#

from my old kernel's driver

short mortar Aug 22, 2024, 9:59 PM

#

flint idol because the command never completes

I think there is a bit you need to set to clear busy on completion

flint idol Aug 22, 2024, 9:59 PM

#

and that bit is.....

short mortar Aug 22, 2024, 10:00 PM

#

In the command list

flint idol Aug 22, 2024, 10:02 PM

#

what if it was just alignment messing this up this entire time

#

but if that were the case qemu would've failed as well

#

LET'S FUCKING GFOOOOOOOOOOOOO

#

and this is why I should never

#

ever

#

look at my old kernel's code

#

and assume it's right

#

what I was doing wrong is I was setting PxCI before setting PxCMD.ST

#

when the spec explicitly says not to

#

NOTE: Never look at the old kernel's code, as it is known by the state of california to cause cancer, birth defects, or other reproductive harm.

#

read and write will be tricky, since it takes in arbitrary buffers

#

which may or may not be paged out

#

and are likely not on a page boundary

#

the last part is probably fine- as the PRDT sets no requirement for physical addresses to be page-algined

#

I'm only worried about the pages not being physically thing

#

contiguous

white mulch Aug 22, 2024, 10:32 PM

#

flint idol contiguous

that's what scatter gather is useful for, as far as I can tell ahci should support it too?

#

so you'd just get the physical page for each virtual page in the buffer and form the list you give it to the controller from those

#

(and make sure that you don't change the backing pages while they are being used by the controller)

flint idol Aug 22, 2024, 10:33 PM

#

white mulch that's what scatter gather is useful for, as far as I can tell ahci should suppo...

I think that's what the prdt does

flint idol Aug 22, 2024, 10:33 PM

#

white mulch (and make sure that you don't change the backing pages while they are being used...

You saying that just saved me days of debugging time

#

The prdt is a list of physical address bases and their size

flint idol Aug 22, 2024, 10:35 PM

#

white mulch (and make sure that you don't change the backing pages while they are being used...

I think I might not use the buffer provided by the user (whether that be the kernel or another driver) as I cannot guarantee this in any way

lean glen Aug 22, 2024, 10:36 PM

#

cant you just idk

#

lock the page

#

so it doesnt get paged out

flint idol Aug 22, 2024, 10:37 PM

#

If I had a page lock, yes

#

I could set pageable to false until the operation completes

#

Of course the ahci driver hangs after port init and before sending of the ata identify command

#

On real hw

#

Unless it's not hanging

#

I have a check if the port type is satapi

#

And if it is the thing aborts

#

For that port

#

But there is no way two ports are satapi

flint idol Aug 22, 2024, 11:04 PM

#

It hangs sending the command

#

I have verified it didn't hang while unmasking irqs or something

#

It hangs while stopping the command engine

#

I'll debug that further in a bit

short mortar Aug 22, 2024, 11:17 PM

#

flint idol the last part is probably fine- as the PRDT sets no requirement for physical add...

It is required to be word-aligned (off the top of my head)

short mortar Aug 22, 2024, 11:21 PM

#

flint idol It hangs while stopping the command engine

I think there is a section in the spec for it
You need to wait for the controller to not be busy (?)

flint idol Aug 22, 2024, 11:27 PM

#

can't wait to get to porting stuff

flint idol Aug 22, 2024, 11:35 PM

#

flint idol can't wait to get to porting stuff

because then I'll be able to replace

OBOS uses ✨ state of the art technology ✨ to do ✨ absolutely nothing ✨
With
OBOS uses ✨ state of the art technology ✨ to do ✨ almost™️ absolutely nothing ✨

vale nymph Aug 22, 2024, 11:38 PM

#

porting stuff feels really rewarding

flint idol Aug 22, 2024, 11:39 PM

#

Unfortunately though, I need to find out why the hell my ahci driver is hanging nooo

flint idol Aug 23, 2024, 12:27 AM

#

PxCMD.CR never goes to zero when the commadn engine is stopped

#

specifically when PxCMD.ST goes zero

flint idol Aug 23, 2024, 10:29 AM

#

decided to debug obos compiled with optimizations instead

#

it's not going very well for the vmm

inland radish Aug 23, 2024, 10:29 AM

#

optimizations enabled is my default mode of testing

#

i only disable optimizations when i need a detailed stack trace and deliberately need it unoptimized

flint idol Aug 23, 2024, 10:30 AM

#

might start doing that

inland radish Aug 23, 2024, 10:30 AM

#

i got so screwed doing things and not testing them with optimizations on

#

back when i started nanoshell

flint idol Aug 23, 2024, 10:31 AM

#

flint idol it's not going very well for the vmm

there's some bug in the vma which causes it to page fault because the rb tree seems to have some funny contents

inland radish Aug 23, 2024, 10:32 AM

#

are you using the freebsd rbtree?

#

or your own impl

flint idol Aug 23, 2024, 10:32 AM

#

indeed

inland radish Aug 23, 2024, 10:32 AM

#

the former?

flint idol Aug 23, 2024, 10:32 AM

#

the former

inland radish Aug 23, 2024, 10:32 AM

#

so probably not a tree bug

#

I bore through with my buggy port of Will's AVL tree implementation for shockingly long

#

but it has a bunch of issues

#

issues I wasn't willing to hunt for

flint idol Aug 23, 2024, 10:34 AM

#

for some reason the kernel isn't getting included in the region list passed to the vmm

flint idol Aug 23, 2024, 10:52 AM

#

for some reason it only gets added to the list when I'm connected with a debugger

#

*it only gets iterated over

#

why is the compiler being weird

#

nvm

#

this seems like it could be some sort of scheduler bug

#

oh nvm

#

it's a vmm bug

#

*vma

#

the rb tree might be corrupted

#

time to enable KASAN and UBSan

#

it caught an actual bug!?!?!?!

#

impossible...

vale nymph Aug 23, 2024, 11:25 AM

#

No way

flint idol Aug 23, 2024, 11:37 AM

#

it seems like a node in the region list passed to the vmm is corrupted

#

well it is

#

idk how

short mortar Aug 23, 2024, 11:45 AM

#

Are you using gcc?

flint idol Aug 23, 2024, 11:45 AM

#

indeed

ashen goblet Aug 23, 2024, 11:45 AM

#

fuck you were right

#

the curse never ends

flint idol Aug 23, 2024, 11:46 AM

#

this is with optimizations on though

#

so it's not that the kernel is fully cursed

ashen goblet Aug 23, 2024, 11:47 AM

#

as several people already mentioned i also like to compile with them on

#

And even sometimes switch them up

#

catches some UB which you otherwise won’t notice (or notice too late)

vale nymph Aug 23, 2024, 11:50 AM

#

flint idol so it's not that the kernel is fully cursed

I would say it is still cursed as its UB now which sucks to debug

flint idol Aug 23, 2024, 11:51 AM

#

fuck it, I'm rewriting that (small, mainly insignificant) part of the kernel

#

I'm replacing the initial virtual memory allocator with a bump allocator

#

I'm giving it 1M

#

since it barely holds any allocs

#

and I don't think anything is ever freed either

#

yay

#

it works

#

rare bump allocator W

#

except for some reason

#

the kernel is slower with opts on that with opts off

#

maybe

#

for some reason, the framebuffer has decided to not be WC anymore

#

it might be on strike for not getting paid enoguh

#

or smth

#

I am also using tcg which might be slowing it down

#

nvm it is a lot faster

#

on kvm

#

nvm it's still broken

#

why the fuck

#

is it double faulting on kvm

#

on cpu 3?

#

ss is 0x00

#

for some reason

#

and rsp is corrupted

#

oh nvm

#

gdb is just confused

#

(but ss is still zero, and the stack pointer is also corrupted)

#

obos curse never leaves nooo

vale nymph Aug 23, 2024, 12:28 PM

#

Obos strikes again

flint idol Aug 23, 2024, 1:15 PM

#

I am no closer to solving this as I was an hour ago

#

Solution:
write bug report and continue on with my life

#

wait I get it on tcg

#

so I can use the qemu log

ashen goblet Aug 23, 2024, 1:16 PM

#

flint idol Solution: write bug report and continue on with my life

you forgot marking the bugreport as WIP ||aka will never be looked at again||

flint idol Aug 23, 2024, 1:17 PM

#

KEKW

#

Servicing hardware INT=0x20
   149: v=20 e=0000 i=0 cpl=0 IP=0008:ffffffff8000005f pc=ffffffff8000005f SP=0010:ffffff00000f0100 env->regs[R_EAX]=0000000000000000
RAX=0000000000000000 RBX=0000000000000000 RCX=0000000000000000 RDX=0000000000000001
RSI=00000000000000aa RDI=ffffff000000311a RBP=0000000000000000 RSP=ffffff00000f0100
R8 =0000000000000000 R9 =0000000000000000 R10=0000000000000000 R11=0000000000000000
R12=0000000000000000 R13=0000000000000000 R14=0000000000000000 R15=0000000000000000
RIP=ffffffff8000005f RFL=00200297 [--S-APC] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0010 0000000000000000 ffffffff 00cf9300 DPL=0 DS   [-WA]
CS =0008 0000000000000000 ffffffff 00af9b00 DPL=0 CS64 [-RA]
SS =0010 0000000000000000 ffffffff 00cf9300 DPL=0 DS   [-WA]
DS =0010 0000000000000000 ffffffff 00cf9300 DPL=0 DS   [-WA]
FS =0010 0000000000000000 ffffffff 00cf9300 DPL=0 DS   [-WA]
GS =0010 ffffff0000002dc0 ffffffff 00cf9300 DPL=0 DS   [-WA]
LDT=0000 0000000000000000 0000ffff 00008200 DPL=0 LDT
TR =0028 ffffff0000002e18 00000070 00408900 DPL=0 TSS64-avl
GDT=     ffffff0000002de0 00000037
IDT=     ffffffff801e8e20 00000fff
CR0=80010011 CR2=0000000000000000 CR3=0000000000006000 CR4=00000020
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
CCS=0000000000000095 CCD=ffffffffffffffff CCO=EFLAGS
EFER=0000000000000d00```
eventually becomes a double fault

#

rsp is waaaaaaay out of bounds

#

at least that's what it looks like

short mortar Aug 23, 2024, 1:46 PM

#

short mortar Are you using gcc?

Why not clang? memedown

flint idol Aug 23, 2024, 1:46 PM

#

too much work

short mortar Aug 23, 2024, 1:48 PM

#

I'd say it was easier to patch than GCC
But there's no wiki articles and such

flint idol Aug 23, 2024, 2:18 PM

#

how

#

the

#

fuck

#

am

#

I

#

failing a bump allocator

#

nooo

short mortar Aug 23, 2024, 2:19 PM

#

Alignment?

flint idol Aug 23, 2024, 2:19 PM

#

nope, memory corruption

short mortar Aug 23, 2024, 2:20 PM

#

~~monolithic kernel L~~

flint idol Aug 23, 2024, 2:20 PM

#

nope, just a skill issue for me

#

oh yeah

#

this is supposed to allocate in pages

flint idol Aug 23, 2024, 3:13 PM

#

I think I might be getting closer to fixing these bugs

#

Fixed

#

but for some reason the ahci driver hangs

#

in debug mode only

#

oh nvm

#

the scheduler hangs

#

obvious solution is to ||make the scheduler lockless||

short mortar Aug 23, 2024, 3:34 PM

#

I have my spinlocks print a stacktrace after 1 million iterations or something

flint idol Aug 23, 2024, 3:34 PM

#

scheduler works lockless therefore it needn't any locks

#

testing the AHCI on real hw again

#

in case some miracle happened between now and yesterday which caused it to work

#

Well it doesn't hang anymore

#

Although it still doesn't work

#

it also identifies two ports as ATAPI, despite there only being one ATAPI port

#

I'm getting invalid builtin

#

PxCI and PxSACT are both 0xffffffff

#

Despite there being no previously issued command

real pecan Aug 23, 2024, 3:58 PM

#

Sounds like ub

flint idol Aug 23, 2024, 4:01 PM

#

as in language UB

#

or AHCI ub

real pecan Aug 23, 2024, 4:03 PM

#

Language

flint idol Aug 23, 2024, 4:03 PM

#

flint idol PxCI and PxSACT are both 0xffffffff

this part

flint idol Aug 23, 2024, 4:04 PM

#

flint idol I'm getting invalid builtin

or this part

real pecan Aug 23, 2024, 4:04 PM

#

Who knows

flint idol Aug 23, 2024, 4:04 PM

#

rebooting to see if it works now

#

(doubt it)

#

Something funny be happening

#

Every port read returns 0xffffffff

#

Maybe I commented out GhcAE = true by accident?

#

nope

#

cpu might be doing some funky shi

#

cpu be doin some funky shi

#

maybe I'm touching the PCI regs wrong

#

causing that to happen

#

that was the case

#

now something else hangs

flint idol Aug 23, 2024, 7:26 PM

#

It works now

flint idol Aug 23, 2024, 7:26 PM

#

flint idol maybe I'm touching the PCI regs wrong

I also fixed this

#

Apparently not every msi device has the capability to be masked

#

Adding a check fixed that

#

I can now remove my overly verbose debug logs

flint idol Aug 23, 2024, 7:44 PM

#

#

I compiled as release

#

time to do a real hw test

#

It boots

#

Sometimes

#

It has decided the hpet no longer has an irq avaliable

vernal chasm Aug 23, 2024, 7:58 PM

#

Challenge: Implement graphics for integrated intel or amd gpus O_O

#

Or just port amd's open source driver or something

flint idol Aug 23, 2024, 7:58 PM

#

No.

vernal chasm Aug 23, 2024, 7:58 PM

#

kekw

flint idol Aug 23, 2024, 7:58 PM

#

Also no.

vernal chasm Aug 23, 2024, 7:58 PM

#

but gwafics sadcat

white mulch Aug 23, 2024, 7:58 PM

#

intel is easier because it has somewhat decent open specs

vernal chasm Aug 23, 2024, 7:58 PM

#

Remember infy did that once

#

@real pecan how much pain was it again?

white mulch Aug 23, 2024, 8:00 PM

#

white mulch intel is easier because it has somewhat decent open specs

also if you just want some vblank interrupts and accelerated page flipping its pretty easy if you rely on the firmware to have set up the mode for you (which means that it must basically be the primary gpu)

real pecan Aug 23, 2024, 8:01 PM

#

vernal chasm <@353593948523986945> how much pain was it again?

Insane

#

Specs are incomplete in a lot of places and have errata

#

Because they aren't really specs but PRMs

white mulch Aug 23, 2024, 8:03 PM

#

also its funny how some of the revisions just randomly lack some important registers

#

though I think the alchemist prms contain all of them again (at least the ones that were missing from the tgl prms)

real pecan Aug 23, 2024, 8:04 PM

#

white mulch also its funny how some of the revisions just randomly lack some important regis...

Yup

real pecan Aug 23, 2024, 8:04 PM

#

white mulch though I think the alchemist prms contain all of them again (at least the ones t...

The what

#

You basically need to read every single one because the information is scattered throughout

white mulch Aug 23, 2024, 8:05 PM

#

real pecan The what

the intel arc discrete gpus

real pecan Aug 23, 2024, 8:05 PM

#

Ah

#

The funny thing is that arc GPUs use a different driver

#

Its built from the ground up to get rid of legacy crap I think

#

But its mostly the same thing iiirc

white mulch Aug 23, 2024, 8:06 PM

#

at least the registers are mostly the same from what I have looked

vernal chasm Aug 23, 2024, 8:11 PM

#

real pecan Insane

lmao

#

I'd kinda like to either implement or port the amd driver for my igpu in my 7950x

#

But it's probably pain there too xD

real pecan Aug 23, 2024, 8:12 PM

#

white mulch at least the registers are mostly the same from what I have looked

Its the same overall architecture

white mulch Aug 23, 2024, 8:13 PM

#

ah

#

with some differences like the south display is on the card itself I assume?

real pecan Aug 23, 2024, 8:13 PM

#

Just e.g. 8 render command streamers instead of 1

#

So its like fast and stuff

real pecan Aug 23, 2024, 8:14 PM

#

white mulch with some differences like the south display is on the card itself I assume?

Yeah all of modesetting stuff is slightly different I think

white mulch Aug 23, 2024, 8:19 PM

#

flint idol Apparently not every msi device has the capability to be masked

were you using the per vector masking?

flint idol Aug 23, 2024, 8:19 PM

#

I was using the mask register, so I think so

white mulch Aug 23, 2024, 8:20 PM

#

yeah that's for it

flint idol Aug 23, 2024, 8:20 PM

#

Oh

white mulch Aug 23, 2024, 8:20 PM

#

I haven't bothered with that and instead just use the enable bit as a mask (though it masks all the msi irqs from the device but most things which I have used msi with only have one anyway)

flint idol Aug 24, 2024, 12:13 AM

#

I'm going to implement reading and writing in a bit

flint idol Aug 24, 2024, 12:50 AM

#

@real pecan when you have the time, could you push the bug fixes you made to hyper for bios

#

where the modesetting wasn't working

flint idol Aug 24, 2024, 12:54 AM

#

flint idol I'm going to implement reading and writing in a bit

after that I will just implement the driver cleanup callback (used for unloading so that the driver can clean up after itself)

#

then I can merge the ahci driver

#

and start on some partition layout things

#

then a FAT driver

#

then a bunch of syscalls

#

then add a target for x86_64-obos-gcc/binutils

#

then mlibc

#

then a bunch of other ports, likely starting with bash

flint idol Aug 24, 2024, 12:56 AM

#

flint idol then a FAT driver

hopefully can start this in one or two days

flint idol Aug 24, 2024, 1:11 AM

#

soon obos will raise to greatness :^)

#

and by that I mean me lifelessly staring at my computer porting stuff nooo

#

@vale nymph what did it take you to port gcc to astral

#

as in, to get it working on astral, and not simply making a target for it

flint idol Aug 24, 2024, 1:18 AM

#

flint idol then a bunch of syscalls

and also (posix) signals

#

and finally, paging out to files/partitions

vale nymph Aug 24, 2024, 1:20 AM

#

flint idol <@215823874615541761> what did it take you to port gcc to astral

not a lot, let me see if I can get a list of the system calls I had

flint idol Aug 24, 2024, 1:21 AM

#

current roadmap:

finish ahci driver
MBR+GPT stuff
FAT driver
Paging out to files and partitions
Posix Signals
TTYs
Pipes (should be ez pz, right?)
Syscalls
Modifications to the kernel to allow for userspace to work (I won't go into detail what these are atm)
Ports

#

it might be cool if I port the third kernel's driver interface to the current kernel

#

so I can run old drivers

#

while certainly not worth it, it'll defintiely be kind of fun

#

I wonder what I'd need to do to implement oDriverInterface, it seems like a nice project, but would probably require tons of kernel callbacks to be useable for drivers

vale nymph Aug 24, 2024, 1:22 AM

#

dq syscall_print
dq syscall_mmap
dq syscall_openat
dq syscall_read
dq syscall_seek
dq syscall_close
dq syscall_archctl
dq syscall_write
dq syscall_getpid
dq syscall_fstat
dq syscall_fstatat
dq syscall_fork
dq syscall_execve
dq syscall_exit
dq syscall_waitpid
dq syscall_munmap
dq syscall_getdents
dq syscall_dup
dq syscall_dup2
dq syscall_dup3
dq syscall_fcntl
dq syscall_chdir
dq syscall_pipe2
dq syscall_isatty
dq syscall_faccessat

#

I had these implemented when I upstreamed gcc

flint idol Aug 24, 2024, 1:23 AM

#

so it's not too hard

#

when I port bash I'd have some of these already

vale nymph Aug 24, 2024, 1:23 AM

#

bash is really ez you just need up until waitpid

#

though it'd be missing some stuff like redirection

vale nymph Aug 24, 2024, 1:26 AM

#

flint idol current roadmap: ``` finish ahci driver MBR+GPT stuff FAT driver Paging out to f...

Also a detail on pipes Ive seen people miss: theres some guarantees on atomicity based on the amount being written

flint idol Aug 24, 2024, 1:27 AM

#

will note that

#

when I'm making pipes

vale nymph Aug 24, 2024, 1:28 AM

#

vale nymph Also a detail on pipes Ive seen people miss: theres some guarantees on atomicity...

AND poll takes that into account
<----- spend a day going after that bug

flint idol Aug 24, 2024, 1:28 AM

#

what does syscall_archctl

#

do

vale nymph Aug 24, 2024, 1:30 AM

#

flint idol what does `syscall_archctl`

Arch specific actions

#

For x86-64 I only really implement set fs base and set gs base

flint idol Aug 24, 2024, 1:30 AM

#

white mulch so you'd just get the physical page for each virtual page in the buffer and form...

marker

flint idol Aug 24, 2024, 1:30 AM

#

vale nymph Arch specific actions

ok

#

has anything used ioctl yet?

vale nymph Aug 24, 2024, 1:32 AM

#

quite a few things

#

some socket stuff, tty stuff, fb stuff, etc

flint idol Aug 24, 2024, 1:33 AM

#

I'll figure those out

#

currently my only way of IO via FDs is with a driver

#

*a kernel module

#

I've implemented all trivial callbacks for the AHCI driver

#

all that is left are the read/write callbacks

flint idol Aug 24, 2024, 2:02 AM

#

populating the PRDTs might be a bit error-prone than I thought

#

I have a maximum of 32 PRDT entries

#

but usually physical addresses in a [contiguous] virtual memory region go from highest to smallest

#

so if someone wants to make 32-pages worth of reads at once, it would break

#

which is only a measly 128 kib of data before it fails

#

I'll just extend the max amount of PRDT entries I can have

#

I'll raise the limit to 1 mib, as that sounds reasonable

#

which extneds the limit to 256 pages

#

*extends

#

aaaaaaaand it hangs

#

I'll fix that later

#

it only happens in release mode

flint idol Aug 24, 2024, 2:11 AM

#

flint idol but usually physical addresses in a [contiguous] virtual memory region go from h...

although I can "merge" physically contiguous virtual memory regions

#

here's some psudeocode of the algorithm that I will use to do this:

OUT: PRDT_list
const MAX_PRDT_COUNT = ...
reg_size = 0
reg_base = 0
prev_phys = 0
for each physical_page in region
  if PRDT_list.size >= MAX_PRDT_COUNT
    return INTERNAL_ERROR
  if !first && (prev_phys + PAGE_SIZE) == physical_page
    reg_size += (bytesLeft > pg_size ? pg_size : bytesLeft)
  else
    PRDT_list.append(reg_base, reg_size)
    reg_base = physical_page
    reg_size = (bytesLeft > pg_size ? pg_size : bytesLeft)
  prev_phys=physical_page
PRDT_list.append(reg_base, reg_size)
assert(PRDT_list.size < MAX_PRDT_COUNT)
return SUCCESS

#

any feedback is welcome

flint idol Aug 24, 2024, 2:24 AM

#

flint idol here's some psudeocode of the algorithm that I will use to do this: ``` OUT: PRD...

this is done after "locking" the pages

flint idol Aug 24, 2024, 2:44 AM

#

I've implemented said algorithm

#

time to test it

flint idol Aug 24, 2024, 3:23 AM

#

#

I can read from the disk now...

#

writing is the exact same

#

but with a different SATA command

#

and I can write as well

#

~~time to test that on real hw~~

#

not writing, that's for sure

#

Reading infinitely hangs it seems

#

Oh wait

#

A read returns all zeroes

#

That's likely a bug with the physical region algorithm thing

#

But I'll debug it tomorrow

#

This is why I gotta test these things

#

If it were found in the fs driver, it could've wrote garbage to some sector if these kind of bugs are not caught

#

It would be funny if I dual boot obos along with windows and debian

#

Wouldn't be hard, I just add a limine chainload entry for hyper

elder hornet Aug 24, 2024, 6:37 AM

#

Very nice progress :D

real pecan Aug 24, 2024, 7:28 AM

#

flint idol <@353593948523986945> when you have the time, could you push the bug fixes you m...

Yeah sure

flint idol Aug 24, 2024, 9:59 AM

#

Ty

real pecan Aug 24, 2024, 10:00 AM

#

i'll also build the release without msvc-rt garbage it depended on

flint idol Aug 24, 2024, 10:02 AM

#

Ok

real pecan Aug 24, 2024, 10:20 AM

#

#

i think this should be good enough

#

as a size and shift guesser

#

will need u to test it

flint idol Aug 24, 2024, 10:21 AM

#

Send

#

Preferably a test image

real pecan Aug 24, 2024, 10:22 AM

#

one sec

#

📎 test.img

flint idol Aug 24, 2024, 10:39 AM

#

@real pecan it booted into the test kernel

real pecan Aug 24, 2024, 10:39 AM

#

ok nice

#

does the screen look ok?

flint idol Aug 24, 2024, 10:39 AM

#

yup

real pecan Aug 24, 2024, 10:39 AM

#

okay thats solved then, nice

flint idol Aug 24, 2024, 10:39 AM

#

it printed out the messages I saw on qemu

#

and the resolution wasn't borked

real pecan Aug 24, 2024, 10:40 AM

#

did u see the warnings printed during boot?

flint idol Aug 24, 2024, 10:40 AM

#

booted too fast

real pecan Aug 24, 2024, 10:40 AM

#

damn oh well

flint idol Aug 24, 2024, 10:40 AM

#

so I couldn't see

real pecan Aug 24, 2024, 10:40 AM

#

also

#

give me the config which hangs bios but erros out on uefi

flint idol Aug 24, 2024, 10:41 AM

#

you mean the one where the bug with old uefi was?

real pecan Aug 24, 2024, 10:41 AM

#

no like

#

remember u tried to escape the string

flint idol Aug 24, 2024, 10:41 AM

#

oh yeah

real pecan Aug 24, 2024, 10:42 AM

#

i wanna see why that doesnt print the error on bios

flint idol Aug 24, 2024, 10:42 AM

#

yeah sure

#

📎 hyper.cfg

#

I don't think I had it in any commits, so I recreated it there

flint idol Aug 24, 2024, 10:43 AM

#

real pecan i wanna see why that doesnt print the error on bios

could be triple faulting while printing the error

real pecan Aug 24, 2024, 10:43 AM

#

also added this to the readme

flint idol Aug 24, 2024, 10:44 AM

#

nope, not a triple fault

real pecan Aug 24, 2024, 10:44 AM

#

lets see

flint idol Aug 24, 2024, 10:45 AM

#

I saw no triple fault in the qemu log

#

unless it's not logging exceptions for some reason

real pecan Aug 24, 2024, 10:45 AM

#

it should

#

#

huh

#

i wonder if that log got stripped out for some reason??

#

what does it look like for u

flint idol Aug 24, 2024, 10:47 AM

#

hangs with no output

real pecan Aug 24, 2024, 10:47 AM

#

can u give me the eip

flint idol Aug 24, 2024, 10:47 AM

#

sure

#

0xb7c2

real pecan Aug 24, 2024, 10:48 AM

#

like there's a dedicated function for that

real pecan Aug 24, 2024, 10:48 AM

#

flint idol 0xb7c2

lets see

flint idol Aug 24, 2024, 10:48 AM

#

inb4 it's not hyper

#

also you can take an obos image

📎 obos.iso

#

to test it

real pecan Aug 24, 2024, 10:49 AM

#

how about i give u this installer instead and then we see if that version prints something instead

#

because it could be a combination of things

flint idol Aug 24, 2024, 10:49 AM

#

ok

real pecan Aug 24, 2024, 10:49 AM

#

📎 hyper_install

#

see if this hangs

flint idol Aug 24, 2024, 10:49 AM

#

ok

real pecan Aug 24, 2024, 10:50 AM

#

oh wait ur using an iso right

flint idol Aug 24, 2024, 10:50 AM

#

yes

real pecan Aug 24, 2024, 10:50 AM

#

you will still have the same stage 2

#

📎 hyper_iso_boot

flint idol Aug 24, 2024, 10:52 AM

#

hang on

#

it's in real mode

real pecan Aug 24, 2024, 10:52 AM

#

wtf

#

do u see anything on the screen at all?

flint idol Aug 24, 2024, 10:53 AM

#

nope

real pecan Aug 24, 2024, 10:53 AM

#

ur probably not using the stage2 i gave u

#

because it doesnt have info logs stripped

#

and theres tons of stuff printed before config parsing

flint idol Aug 24, 2024, 10:54 AM

#

nope

#

still nothing

real pecan Aug 24, 2024, 10:55 AM

#

wtf

flint idol Aug 24, 2024, 10:55 AM

#

btw it's in a cli hlt loop

#

at 0xfb7c1

#

when I used the $pc register instead of $eip

real pecan Aug 24, 2024, 10:55 AM

#

and that somehow depends on config?

flint idol Aug 24, 2024, 10:55 AM

#

seems like it

real pecan Aug 24, 2024, 10:56 AM

#

i still dont understand how there isnt a log

#

because if it depends on config its very late in the boot

#

can u produce normal disks instead of an iso?

#

that way i can at least install to it myself

flint idol Aug 24, 2024, 10:56 AM

#

I can, just give me a bit

real pecan Aug 24, 2024, 10:56 AM

#

sure

flint idol Aug 24, 2024, 10:56 AM

#

as I don't neccessarily have a target for it in obos' build system

real pecan Aug 24, 2024, 10:57 AM

#

inb4 no repro on a normal disk

#

oh wait also

#

did u try moving the qemu window

flint idol Aug 24, 2024, 10:58 AM

#

no

#

still nothing

real pecan Aug 24, 2024, 10:59 AM

#

yup

#

that was it

#

try resizing it or moving around

#

literally the iso u sent

#

its an old qemu bug

#

if u halt the cpu too quickly the screen is not updated

flint idol Aug 24, 2024, 11:00 AM

#

I'm using qemu 9.0.50

real pecan Aug 24, 2024, 11:00 AM

#

well it might not have been fixed on linux ig

flint idol Aug 24, 2024, 11:00 AM

#

time to switch back to windows ultrameme

real pecan Aug 24, 2024, 11:00 AM

#

does resizing the window also not work?

flint idol Aug 24, 2024, 11:01 AM

#

nope

real pecan Aug 24, 2024, 11:01 AM

#

#

works with wsl as well

#

okay how about i enable e9 logging and we see what it outputs on your qemu

flint idol Aug 24, 2024, 11:02 AM

#

sure

real pecan Aug 24, 2024, 11:03 AM

#

📎 hyper_iso_boot

flint idol Aug 24, 2024, 11:03 AM

#

that prints out the logs

real pecan Aug 24, 2024, 11:03 AM

#

does it print out the error as well?

flint idol Aug 24, 2024, 11:03 AM

#

yup

real pecan Aug 24, 2024, 11:03 AM

#

yup so there goes that bug

#

okay the last and the biggest thing left is uefi memory allocator

#

that one is gonna be a pain but ill try to get it done

flint idol Aug 24, 2024, 11:04 AM

#

ok ty

real pecan Aug 24, 2024, 11:05 AM

#

np

#

oh u could also try doing like ctrl+alt+2 and then ctrl+alt+1 again

#

but maybe taht default qemu frontend is just bugged

flint idol Aug 24, 2024, 11:08 AM

#

accidentally switched to tty2

#

and needed to restart the X server

#

or whatever

real pecan Aug 24, 2024, 11:08 AM

#

lmao

#

the tty switching is a similar key combo

flint idol Aug 24, 2024, 11:08 AM

#

ik

real pecan Aug 24, 2024, 11:08 AM

#

but qemu has ttys as well

flint idol Aug 24, 2024, 11:09 AM

#

I use it when compiling qemu and stuff on my computer

#

to save around 3 gigabytes of memory

real pecan Aug 24, 2024, 11:09 AM

#

are u that limited on ram

flint idol Aug 24, 2024, 11:09 AM

#

8 gib is not that much

real pecan Aug 24, 2024, 11:09 AM

#

ah yeah its not

flint idol Aug 24, 2024, 11:12 AM

#

real pecan but maybe taht default qemu frontend is just bugged

still happens, even on the sdl backend

real pecan Aug 24, 2024, 11:13 AM

#

thats annoying

#

idk how u get qemu to flush it properly

flint idol Aug 24, 2024, 11:13 AM

#

also just found out qemu has a curses backend

#

which also isn't flushed???

real pecan Aug 24, 2024, 11:14 AM

#

what if u disable kvm

flint idol Aug 24, 2024, 11:14 AM

#

I'll see

flint idol Aug 24, 2024, 11:14 AM

#

flint idol also just found out qemu has a curses backend

oh nvm it's just shit

#

still nothing

real pecan Aug 24, 2024, 11:15 AM

#

btw

#

what if u run this in like

#

vbox

flint idol Aug 24, 2024, 11:16 AM

#

I tried, and I get logs

real pecan Aug 24, 2024, 11:16 AM

#

ah

flint idol Aug 24, 2024, 11:16 AM

#

what is egl-headless

real pecan Aug 24, 2024, 11:16 AM

#

one more thing u could try is

#

-vnc

#

and then use like remmina or something to look at the screen

#

-vnc localhost:0

flint idol Aug 24, 2024, 11:18 AM

#

still nothing

real pecan Aug 24, 2024, 11:19 AM

#

bruh LULW

#

my wsl is qemu 8.1.1 and my pc is 8.1

#

maybe they fucked it up again

#

lemme build the lastest qemu

#

if it doesnt output anything ill bisect the bug

flint idol Aug 24, 2024, 11:45 AM

#

I think I have a bug with my physical region populate thing

#

there is a possibility it overflows into arbitrary physical memory

#

if the current size makes it so that it crosses page boundaries

#

simple solution should be to:

size_t bytesInPage = (addr % pg_size);

#

and use that

#

to deduce the size of the region at hand

flint idol Aug 24, 2024, 11:48 AM

#

flint idol simple solution should be to: ``` size_t bytesInPage = (addr % pg_size); ```

and if that is zero, it is rounded up the the page size

real pecan Aug 24, 2024, 11:52 AM

#

yeah no

#

i think its either a bug in your window manager or a bug in your specific qemu

flint idol Aug 24, 2024, 11:53 AM

#

try:

qemu-system-x86_64 \
-drive file=obos.iso,format=raw \
-m 1G \
-gdb tcp:0.0.0.0:1234 -S \
-M q35 \
-cpu host \
-accel kvm \
-debugcon file:/dev/stdout \
-monitor stdio \
-smp cores=4,threads=1,sockets=1 \
-M smm=off \
-d int \
-D qemu_log.txt```

#

maybe that somehows changes things

real pecan Aug 24, 2024, 11:57 AM

#

smm=off

#

ig bios printing doesnt work without it

flint idol Aug 24, 2024, 11:58 AM

#

guess so

real pecan Aug 24, 2024, 11:58 AM

#

does it work now?

flint idol Aug 24, 2024, 11:58 AM

#

yup

real pecan Aug 24, 2024, 11:58 AM

#

makes sense

flint idol Aug 24, 2024, 12:06 PM

#

flint idol simple solution should be to: ``` size_t bytesInPage = (addr % pg_size); ```

that calculation needed to be pg_size - (addr % pg_size)

flint idol Aug 24, 2024, 12:26 PM

#

I love it when I crash qemu with my ahci driver

real pecan Aug 24, 2024, 12:37 PM

#

i think i found a cheap way to work around that uefi bug

#

my mm_fixup function is advanced enough already to resolve range conflicts, ill just inject the custom type allocations at the end and have it resolve the conflicts against LoaderData on its own

flint idol Aug 24, 2024, 12:38 PM

#

ok

real pecan Aug 24, 2024, 12:38 PM

#

will se if that works

flint idol Aug 24, 2024, 12:39 PM

#

I think I can read sectors now without corrupting random regions of memory

#

and without crashing qemu

real pecan Aug 24, 2024, 12:40 PM

#

nice

real pecan Aug 24, 2024, 12:59 PM

#

yay it works lmao

#

most hacky solution but its technically the best one

#

just shove the entries into the uefi memory map and pretend they were there

#

this might break because its hacky af but can u give it a try

📎 hyper_uefi

flint idol Aug 24, 2024, 1:02 PM

#

is it the thing I pass to xorriso

real pecan Aug 24, 2024, 1:02 PM

#

its the BOOTX64.EFI

flint idol Aug 24, 2024, 1:03 PM

#

btw how do I make the image to pass in there

#

I just made a giant fat32 image

#

and threw in the bootx64.efi

real pecan Aug 24, 2024, 1:03 PM

#

how do u do uefi boot right now

#

just replace the file with this one

real pecan Aug 24, 2024, 1:04 PM

#

flint idol I just made a giant fat32 image

thats correct

flint idol Aug 24, 2024, 1:04 PM

#

how do make the fat32 image not as giant

#

otherwise mkfs.fat gives me a warning

#

when the fat image is too small

real pecan Aug 24, 2024, 1:05 PM

#

i just do this

def make_fat(raw_fs_path: str, _: int, force_fat32: bool) -> None:
    cr_args = ["mformat", "-i", raw_fs_path]
    if force_fat32:
        cr_args.append("-F")

    subprocess.check_call(cr_args)

flint idol Aug 24, 2024, 1:06 PM

#

I didn't know there was a commadn called mformat

real pecan Aug 24, 2024, 1:06 PM

#

yeah its part of mtools

#

u can copy files onto the image with it too

#

def fat_recursive_copy(raw_fs_path: str, file_path: str) -> None:
    subprocess.check_call([
        "mcopy", "-Q", "-i", raw_fs_path, "-s", file_path, "::"
    ])

flint idol Aug 24, 2024, 1:07 PM

#

Too few clusters for 32 bit fat

#

when I run it with BOOTX64.efi

real pecan Aug 24, 2024, 1:07 PM

#

make a fat12?

flint idol Aug 24, 2024, 1:08 PM

#

ok then

real pecan Aug 24, 2024, 1:08 PM

#

flint idol Too few clusters for 32 bit fat

is that like

#

mformat output?

flint idol Aug 24, 2024, 1:08 PM

#

yes

real pecan Aug 24, 2024, 1:08 PM

#

yeah fat32 requires at least like 2 megs i think

#

just dont force fat32

flint idol Aug 24, 2024, 1:09 PM

#

ok

flint idol Aug 24, 2024, 1:30 PM

#

I tried reading the entire disk at once

#

and failed

#

got all zeroes

#

I think the irq is being issued too early

flint idol Aug 24, 2024, 2:04 PM

#

the read works on vbox

flint idol Aug 24, 2024, 3:25 PM

#

for some reason the lapic timer breaks on uefi

#

on kvm

flint idol Aug 24, 2024, 4:33 PM

#

@real pecan can confirm that the uefi bug fix works where it was previously broken

#

Lemme reword that

#

The bug fix fixed the bug

real pecan Aug 24, 2024, 4:35 PM

#

Nice

#

I didn't expect the workaround to be like 30 lines of code

#

But it was a genius idea

#

Essentially I just pretend the EFI put this allocation in the memory map twice and let my memory map sanitizer take care of collision resolution

flint idol Aug 24, 2024, 4:38 PM

#

real pecan I didn't expect the workaround to be like 30 lines of code

neither did I

#

anyway, the ahci driver doesn't want to do a read on real hw

#

because for some reason, the prdts are never filled

#

if it were an error, then it would've returned failure

#

but it said success

#

so rather it's lying to me

#

or there is just nothing in the PRDT

flint idol Aug 24, 2024, 4:47 PM

#

flint idol because for some reason, the prdts are never filled

That's because the port doesn't exist

#

As in, identify ata was never sent to it

#

I was able to fix that

#

But now the driver hangs on the read

real pecan Aug 24, 2024, 6:49 PM

#

flint idol <@353593948523986945> can confirm that the uefi bug fix works where it was previ...

Like all PCs where it was broken right not just one

flint idol Aug 24, 2024, 6:50 PM

#

I'll see if the bug is fixed on the other computer later

real pecan Aug 24, 2024, 6:50 PM

#

Thanks

#

Well I gave u a hacky proof of concept so a retest will be needed eventually anyway

flint idol Aug 24, 2024, 8:02 PM

#

@real pecan it works on the other test subject

real pecan Aug 24, 2024, 8:03 PM

#

nice

flint idol Aug 24, 2024, 8:04 PM

#

I have a bug with my kernel though where the kernel hangs after uACPI init

flint idol Aug 24, 2024, 9:12 PM

#

I'm adding this to my limine config for my computer

vernal chasm Aug 24, 2024, 9:39 PM

#

I'm sorry what O_O

#

using limine to boot with hyper?

flint idol Aug 24, 2024, 9:39 PM

#

Yes

vernal chasm Aug 24, 2024, 9:39 PM

#

but why?

flint idol Aug 24, 2024, 9:39 PM

#

I use the chainload

flint idol Aug 24, 2024, 9:40 PM

#

vernal chasm but why?

Hyper has no boot manager

vernal chasm Aug 24, 2024, 9:40 PM

#

and?

flint idol Aug 24, 2024, 9:40 PM

#

This is on my computer

vernal chasm Aug 24, 2024, 9:40 PM

#

Practically all uefi bioses have that built in...

#

By just holding down a function key

flint idol Aug 24, 2024, 9:40 PM

#

Too much work

vernal chasm Aug 24, 2024, 9:40 PM

#

bruh

#

kekw

flint idol Aug 24, 2024, 9:40 PM

#

Also mine acts funny

vernal chasm Aug 24, 2024, 9:40 PM

#

hold function key, select drive and you're good

#

lmao

flint idol Aug 24, 2024, 9:40 PM

#

When I use efibootmgr

flint idol Aug 24, 2024, 9:40 PM

#

vernal chasm hold function key, select drive and you're good

This is all on the same drive

vernal chasm Aug 24, 2024, 9:41 PM

#

It should also support multi volume drives

flint idol Aug 24, 2024, 9:41 PM

#

Not mine

vernal chasm Aug 24, 2024, 9:41 PM

#

tf

#

who made yours then?

flint idol Aug 24, 2024, 9:42 PM

#

rn_image_picker_lib_temp_f9bd97c7-ee0a-41d0-bebd-3437180c1ad3.jpg

vernal chasm Aug 24, 2024, 9:42 PM

#

Ah I see

#

But shouldn't they be using the american megatrends one tho?

#

At least my semi old acer laptop I had used that

real pecan Aug 25, 2024, 10:29 AM

#

@flint idol can u give me the model of your computer with that shitty bios

#

the one that zeroes x components

#

ill mention it in the commit message

real pecan Aug 25, 2024, 11:35 AM

#

I found this trash btw https://github.com/tianocore/edk2/commit/10fe0d814add860a1040e648b1f5782c0de350e6

GitHub

Fix crash in GetMemoryMap() when memory type 0x80000000-0xFFFFFFFF ...

…s allocated by OS loader

Signed-off-by: niruiyu
Reviewed-by: lgao4
Reviewed-by: hhtian

git-svn-id: https://edk2.svn.sourceforge.net/svnroot/edk2/trunk/edk2@11774 6f19259b-4bc3-4df7-8a09-76579488...

#

Ill make a separate thread for hyper instead of littering ur thread ig

real pecan Aug 25, 2024, 12:29 PM

#

@flint idol give this a try once ur back

📎 hyper_uefi

#

this contains a non hacky proper implementation of uefi memory types workaround

flint idol Aug 25, 2024, 3:06 PM

#

ok

#

@real pecan boots on one test aubject

#

*subject

#

And so does test subject two

#

BTW do you know of anything that could be causing this in my kernel api

#

As in, the error

#

Oh and gpes work on this device

#

I'll just read the asl

#

Method (ESEL, 0, Serialized)
{
    If (((XHCI == 0x02) || (XHCI == 0x03)))
    {
        PR3 &= 0xFFFFFFC0
        PR2 &= 0xFFFF8000
        XUSB = Zero
        XRST = Zero
    }
}```

real pecan Aug 25, 2024, 3:18 PM

#

flint idol As in, the error

One of your read or write handlers returning this

flint idol Aug 25, 2024, 3:19 PM

#

I assume it'd be a PCI rw handler?

real pecan Aug 25, 2024, 3:19 PM

#

Check

#

Idk which object these fields are part of

real pecan Aug 25, 2024, 3:19 PM

#

flint idol And so does test subject two

Nice, that's cool to hear

flint idol Aug 25, 2024, 3:20 PM

#

my kernel api assumes any error with the arch-specific pci read/write is an invalid argument

#

which is correct

#

because that's all it seems to return

#

obos_status DrvS_ReadPCIRegister(pci_device_location loc, uint8_t offset, size_t accessSize, uint64_t* val)
{
    if (!val)
        return OBOS_STATUS_INVALID_ARGUMENT;
    switch (accessSize)
    {
        case 1:
            *val = pciReadByteRegister(loc.bus, loc.slot, loc.function, offset);
            break;
        case 2:
            *val = pciReadWordRegister(loc.bus, loc.slot, loc.function, offset);
            break;
        case 4:
            *val = pciReadDwordRegister(loc.bus, loc.slot, loc.function, offset);
            break;
        default:
            return OBOS_STATUS_INVALID_ARGUMENT;
    }
    return OBOS_STATUS_SUCCESS;
}```

#

that's all it is

real pecan Aug 25, 2024, 3:21 PM

#

So add printfs and see what's failing

real pecan Aug 25, 2024, 3:29 PM

#

real pecan <@841762704766140486> can u give me the model of your computer with that shitty ...

Also this

flint idol Aug 25, 2024, 3:29 PM

#

sure

#

I just need to find it

#

when I'm done adding logs to my kernel api I will

real pecan Aug 25, 2024, 3:31 PM

#

Sure thanks

flint idol Aug 25, 2024, 3:31 PM

#

rn_image_picker_lib_temp_54e46a14-d9b0-4f40-90b4-2f2fe8188c15.jpg

rn_image_picker_lib_temp_c27d0c21-b4bc-4082-a40f-762ed5d614b3.jpg

#

Oops

#

I added the logs

#

But never added anything that said where it failed

#

now that's why I like macros in C

#

I can just

#define OBOS_Debug(msg) OBOS_Debug(__FILE__ ":%d in %s. %s\n", __func__, msg);

#

temporarily instead of changing each line with the debug print

real pecan Aug 25, 2024, 3:36 PM

#

flint idol

Thanks

flint idol Aug 25, 2024, 3:48 PM

#

I fixed the bug

#

But it still hangs after uacpi init

real pecan Aug 25, 2024, 3:48 PM

#

flint idol I fixed the bug

What was it

flint idol Aug 25, 2024, 3:49 PM

#

Previously when I was making the pci code, I copied and pasted the read function for the write function

#

And changed some lines

#

but forgot to remove the line that checked if val was nullptr

#

because val was now a uint64_t

real pecan Aug 25, 2024, 3:49 PM

#

Oh

flint idol Aug 25, 2024, 3:50 PM

#

flint idol ```asl Method (ESEL, 0, Serialized) { If (((XHCI == 0x02) || (XHCI == 0x03))...

it probably failed on XUSB = Zero

#

idk whether it's a PCI reg or not, but it certainly was set to zero

#

It seems to have been infinite gpes

#

Now it hangs in the ahci driver while waiting for an irq

#

After I send the read command

real pecan Aug 25, 2024, 3:56 PM

#

flint idol idk whether it's a PCI reg or not, but it certainly was set to zero

Look up what that field is, but probably is

flint idol Aug 25, 2024, 3:57 PM

#

Ok

flint idol Aug 25, 2024, 3:57 PM

#

flint idol Now it hangs in the ahci driver while waiting for an irq

This is weird because I get irqs for IDENTIFY ATA

#

On both devices

#

rn_image_picker_lib_temp_56a19c2b-1b85-4bcb-8eb9-faa6b9cadd01.jpg

real pecan Aug 25, 2024, 4:41 PM

#

flint idol

ok done

flint idol Aug 25, 2024, 4:47 PM

#

flint idol This is weird because I get irqs for IDENTIFY ATA

after some changes, I now receive an IRQ before an IDENTIFY_ATA command is completed, causing a hang

#

specifically, the irq is received, but I get it before the bit in PxCI is cleared

#

so nothing is ever signalled

#

note that this is after I changed the send command function to set the prdt entries' i field to zero

flint idol Aug 25, 2024, 5:15 PM

#

I printed out PxIS to see that a PIO Setup FIS was received

#

except I never send a PIO Setup fis

flint idol Aug 25, 2024, 6:19 PM

#

I can read from disk on one piece of hw

#

#

One device hangs

ashen goblet Aug 25, 2024, 6:37 PM

#

did you fix the driver?

flint idol Aug 25, 2024, 6:37 PM

#

For most devices yes

ashen goblet Aug 25, 2024, 6:39 PM

#

what was it?

flint idol Aug 25, 2024, 7:48 PM

#

ashen goblet what was it?

I believe I was setting HBA->is to the wrong value after IRQs

#

so the irq status wasn't being cleared properly, causing no IRQ

#

I think I know how to fix the bug

#

instead of waiting on an irq

#

during the initial IDENTIFY ATA phase

#

I just wait until the bit in PxCI goes zero

#

I now get random bullshit from IDENTIFY ATA

#

On that device

#

rn_image_picker_lib_temp_40d6d259-0154-409a-a514-655d1a72ea3d.jpg

#

while (HBA->ports[port->hbaPortIndex].ci & data.internal.cmdSlot)

#

oopsie

vernal chasm Aug 25, 2024, 8:02 PM

#

flint idol

flint idol Aug 25, 2024, 8:03 PM

#

that sticker was my inspiration for my debug logs

#

while debugging

#

After changing that line

#

I still get random bs from identify ata

#

But I feel like I recognize this data though

#

From my previous ahci driver

flint idol Aug 25, 2024, 8:13 PM

#

flint idol I still get random bs from identify ata

and for some reason it suddenly is able to receive IRQs

#

it seems like some ahci controllers are more picky on the values set

#

specifically alignment

flint idol Aug 25, 2024, 8:56 PM

#

I fixed the bug

#

It was because I was assuming some identify ata fields exist

#

But I don't get any irqs

#

rn_image_picker_lib_temp_3f159615-87ee-4dc4-a8eb-7e5194e5b935.jpg

#

Why could this be?

#

well if the prdt entry's i field is set to one

#

I should at least get an irq before the command is completely done

#

but I don't thonk

#

I also have a bug with the read/write commands

#

because of limits with the ATA spec

#

I can only read up to 65536 sectors at once

#

but those functions take in a uint64_t

#

Idfk what I did

#

But it works

#

rn_image_picker_lib_temp_64ead91b-193d-4ddb-bd9a-5efbc729f2c1.jpg

#

wtg

#

*wtf

#

when I remove the logs

#

it stops working on qemu

#

I added back a log

#

*two logs

#

and it works

#

without this log:

OBOS_Debug("bonjour HBA->is: 0x%08x\n", HBA->is);

#

in the irq checker

#

it stops working

#

when I remove two other logs

#

it works without that one

#

I made the SendCommand function thread-safe

#

was pretty trivial, just needed to add a bitmask for pending commands

#

then I threw a mutex on that

flint idol Aug 25, 2024, 9:32 PM

#

flint idol current roadmap: ``` finish ahci driver MBR+GPT stuff FAT driver Paging out to f...

marker

#

https://github.com/OBOS-dev/obos/pull/22

GitHub

AHCI driver by oberrow · Pull Request #22 · OBOS-dev/obos

That's all it is. An AHCI driver.

#

why is there 2k changes

#

time to merge

#

and do the partition layout thing

#

I think I'll add that code to some file in the root of the kernel directory

#

I'm screensharing that in voice one

#

if anyone would want to watch the gruesome task of me writing partition code

flint idol Aug 25, 2024, 10:20 PM

#

the mbr code was pretty straight forward

#

I need to write gpt code

#

and also a function that actually calls these and stuff

#

like partprobe

#

basically it first will try to identify a GPT partition table

#

if that fails, it tries to use MBR

#

otherwise it makes a raw partition spanning the entire disk

#

also obos has 100 source files now

#

including libraries

#

183 files including headers

#

199 files

#

to the UEFI spec I go

flint idol Aug 25, 2024, 11:03 PM

#

I practically copy pasted my old kernel's GPT implementation

flint idol Aug 26, 2024, 12:30 AM

#

I have made it so after PnP, you have an option to wait for the drivers to finish initialization

#

I also implemented the thing

#

that probes a drive for partitions

#

first checks gpt

#

if that fails

#

it checks mbr

#

if that fails, it registers a raw partition ranging the entire drive

#

wtf

#

how is a NMI being masked

#

whatever

#

the VFS works with block devices after changing a small bug with seek

#

except nRead is zero...

#

might be a bug with the ahci driver

#

it was a bug with a function in the ahci driver

#

responsible for reading

#

well that's weird...

#

the GPT checksums don't match

#

(here we go again)

#

on the iso it is 0x97A61096

#

I'll see what I calculate

#

oops

#

I was setting the wrong checksum field to zero

#

shit

#

the ahci driver has decided to fail on me

#

it's returning a read of all zeroes

#

it seems to be a bug with the populate_physical_regions function

#

since each physical region is one physical page behind its actual value

#

I think I see the bug

#

now it doesn't want to read anything

#

ahci_populate_sglist ahci(0x558a3b0f9b50)[0]
ahci_dma_prepare_buf ahci(0x558a3b0f9b50)[0]: prepare buf limit=31744 prepared=28672
ahci_cmd_done ahci(0x558a3b0f9b50)[0]: cmd done```

#

well quite obviously it didn't prepare shit

#

because the buffer is full of zeroes

#

despite the data there not being full of zeroes

flint idol Aug 26, 2024, 1:47 AM

#

flint idol ``` ahci_populate_sglist ahci(0x558a3b0f9b50)[0] ahci_dma_prepare_buf ahci(0x558...

"cmd done" my ass

#

if the command were done I would've

#

gotten

#

fucking

#

data

#

in my

#

fucking buffer

flint idol Aug 26, 2024, 1:53 AM

#

flint idol ``` ahci_populate_sglist ahci(0x558a3b0f9b50)[0] ahci_dma_prepare_buf ahci(0x558...

note that this is a read of 62 sectors

#

over 7 pages

#

flint idol Aug 26, 2024, 2:17 AM

#

that is a raw partition

#

since sdb is filled with 0xaa

#

while I was testing before

#

it finds 185 partitions...

#

(there are not 185 partitions in the disk image)

vernal chasm Aug 26, 2024, 2:51 AM

#

flint idol how is a NMI being masked

That sounds like maskable non maskable interrupts kekw

flint idol Aug 26, 2024, 2:59 AM

#

I got GPT partitioning working

#

(note that the offsets are in bytes, not sectors)

#

and I made sure it matched the partitions in gdisk

#

time to test on my hw

#

and hope I don't accidentally corrupt anything

#

btw this doesn't work on qemu because the AHCI emulation seems buggy on qemu

#

It only identifies 4 partitions out of my 7

flint idol Aug 26, 2024, 11:04 AM

#

flint idol It only identifies 4 partitions out of my 7

it probably isn't the data being read unproperly

#

because I check the checksum

#

of the partition table

#

maybe that's failing and falling back to the protective MBR

#

but it wouldn't have 4 partitions then

empty kernel Aug 26, 2024, 11:07 AM

#

With GPT, mbr only has 1 partition

flint idol Aug 26, 2024, 11:07 AM

#

yeah

flint idol Aug 26, 2024, 11:43 AM

#

I just wrote some UUID code

#

and I need to provide some way for fs drivers to manipulate partitions

#

as in, use a partition as a filesystem

#

that it loads

#

if that makes sense

#

I'll make two APIs take in a partition

#

and let the filesystem driver do the hard work

#

I'll make list_dir and path_search take in a partition

#

I'll also add an API which probes a partition for an fs in the function table

#

also one to initialize the fs there

#

actually instead of a partition it'll take in a vnode

#

or maybe a dirent to make opening the vnode easier

flint idol Aug 26, 2024, 12:25 PM

#

I really did not think file mapping through...

#

shared mappings are broken

#

unless they aren't and I can't read

flint idol Aug 26, 2024, 3:44 PM

#

it's once again broken with optimizations

#

this time, within a driver, it cannot find a mapped page

#

in the page tree

#

despite it existing

flint idol Aug 26, 2024, 4:04 PM

#

flint idol this time, within a driver, it cannot find a mapped page

this only happens with pages within the kernel binary

flint idol Aug 26, 2024, 4:45 PM

#

flint idol I'll make `list_dir` and `path_search` take in a `partition`

I did that hours ago

#

partitions basically work

#

I just needa fix my UUID functions

flint idol Aug 26, 2024, 5:50 PM

#

done

flint idol Aug 26, 2024, 5:50 PM

#

flint idol It only identifies 4 partitions out of my 7

I also fixed this bug

#

and added a workaround for the bug where large writes return all-zero

#

on qemu

#

size_t i = 0;
for (; i < nSectorsForPartitionTable; i += 4)
    Vfs_FdRead(desc, buf+i*blkSize, blkSize*4, nullptr);
if (i > nSectorsForPartitionTable)
    Vfs_FdRead(desc, buf+i*blkSize, blkSize*(i-nSectorsForPartitionTable), nullptr);```

flint idol Aug 26, 2024, 7:22 PM

#

reading from AHCI occasionally hangs

#

but obviously that not very acceptable

#

well this is weird

#

the event is signalled- but the thread waiting on it is never unblocked

#

even the thread itself recognizes the fact that it is waiting on 1 object, and 1 object signalled it

#

yet it still says it's blocked

#

maybe it is a race condition

#

between the signal function and the wait function

#

I'll protect waitable objects with a mutex/spinlock

#

since mutexes are waitable, I must use a spinlock

#

I've made all waitable objects need to be acquired at IRQL_PASSIVE

#

and they are also protected by a spinlock

flint idol Aug 26, 2024, 7:54 PM

#

flint idol I've made all waitable objects need to be acquired at `IRQL_PASSIVE`

I think this needs to be changed to IRQL_DISPATCH

#

I have made the AHCI driver a lot more stable

#

as it now doesn't hang

#

as much

flint idol Aug 26, 2024, 8:06 PM

#

flint idol I have made the AHCI driver a lot more stable

and probably some future drivers as well

#

because this forces me to use DPCs and lower irqls more

vernal chasm Aug 26, 2024, 8:06 PM

#

flint idol as much

it froze right before this message amirit?

flint idol Aug 26, 2024, 8:06 PM

#

KEKW

#

yes

vernal chasm Aug 26, 2024, 8:06 PM

#

lmao

flint idol Aug 26, 2024, 8:09 PM

#

anyway

#

now that I got partitions working

#

actually I want to test MBR a bit first

#

actually nah who even uses that shit

vernal chasm Aug 26, 2024, 8:10 PM

#

flint idol actually nah who even uses that shit

your mom

#

:>

#

But actually a lot of people still use MBR...

flint idol Aug 26, 2024, 8:11 PM

#

well it probably works

#

I think I saw it working when it was falling back to the (protective) MBR when my GPT code was broken

vernal chasm Aug 26, 2024, 8:11 PM

#

O_O

#

But cool I guess

flint idol Aug 26, 2024, 8:12 PM

#

time to work on the 32-bit ms dos filesystem driver

#

(I'm not saying the name this time)

#

just merged the partition layouts

#

theoretically a 32-bit ms dos fs driver should be easy

#

might be interesting to implement an exMSDOSFilesystem driver

#

@barren merlin how much more effort is to to support FAT12/FAT16 alongside FAT32

barren merlin Aug 26, 2024, 8:25 PM

#

not much

#

plus a correct implementation has to support all three

#

because FAT type is defined by the amount of clusters

flint idol Aug 26, 2024, 8:26 PM

#

right

#

does FAT12/FAT16 support the lfs extensions?

barren merlin Aug 26, 2024, 8:26 PM

#

yes

flint idol Aug 26, 2024, 8:26 PM

#

flint idol does FAT12/FAT16 support the lfs extensions?

*lfn

barren merlin Aug 26, 2024, 8:26 PM

#

directory entry format is identical

#

only difference is location of root directory

flint idol Aug 26, 2024, 8:27 PM

#

I think the difference is in the BPB and root directory

barren merlin Aug 26, 2024, 8:27 PM

#

and lack of the FS info sector

#

and ofc width of FAT entries

#

FAT12 is a bit annoying to implement in that regard, but not too much

flint idol Aug 26, 2024, 8:27 PM

#

ok, ty for the help

vernal chasm Aug 26, 2024, 8:31 PM

#

flint idol <@535830483863470110> how much more effort is to to support FAT12/FAT16 alongsid...

Also do exFAT alongside that O_O

flint idol Aug 26, 2024, 8:32 PM

#

vernal chasm Also do exFAT alongside that O_O

TODO:

vernal chasm Aug 26, 2024, 8:32 PM

#

oof

flint idol Aug 26, 2024, 8:32 PM

#

basically everything is different from what I could see

flint idol Aug 26, 2024, 8:55 PM

#

I have a fat date, again!
||c typedef struct fat_date { uint8_t day : 5; uint8_t month : 4; uint8_t year1980 : 7; } OBOS_PACK fat_date;||

flint idol Aug 26, 2024, 10:44 PM

#

after many procrastination (accidentally opened my phone)

#

I will continue working on the fat driver

flint idol Aug 26, 2024, 11:02 PM

#

after a minor pause

#

I will now continue

#

working on the fat driver

#

I have all the structs in place

#

I just need to add a new function to the function table

#

that's simply:

// Probes a vnode to see if it contains this filesystem
// Returns true if the filesystem that this driver represents can load the filesystem in this vnode.
// Mustn't be nullptr except for the InitRD driver.
bool(*probe)(vnode* vn);```

#

the fat driver will probably store a bit of info for the vnode in some table

#

if the probe returns true

flint idol Aug 26, 2024, 11:22 PM

#

flint idol the fat driver will probably store a bit of info for the vnode in some table

or maybe the entire FAT

#

you never know

#

oh god

The next data structure that is important is the FAT itself. What this data structure does is define a
singly linked list of the “extents” (clusters) of a file.

#

hol' up

#

those better not be the directory entries

flint idol Aug 27, 2024, 12:20 AM

#

is the FAT just a bitfield

#

of each allcoated cluster

#

if so then that's pretty convinient

#

basically just needa write a bitmap allocator for writing

flint idol Aug 27, 2024, 12:38 AM

#

fat12 sure do be weird...

#

We now access the FAT entry as a WORD just as we do for FAT16, but if the cluster number is EVEN, we only want the low 12-bits of the 16-bits we fetch; and if the cluster number is ODD, we only want the high 12-bits of the 16-bits we fetch.

flint idol Aug 27, 2024, 12:42 AM

#

flint idol > We now access the FAT entry as a WORD just as we do for FAT16, but if the clus...

shouldn't be a problem though

#

I need to implement initialization of the volume cache

#

I'm 50% done

#

I just need to implement the thing that searches the directory entries

#

and adds the cache entries

flint idol Aug 27, 2024, 2:01 AM

#

flint idol and adds the cache entries

that should be done

#

whether it works or not?

#

probably not first try

#

I need to implement an unimplemented feature before I continue

#

basically just need to load modules as specified on the command line after PnP

#

ASAN Violation at ffffffff8005b51e while trying to write 8 bytes from 0xffff80003958e398 (Hint: Use of memory block after free).```
in the vmm

#

fuck

#

fixed it

#

why

#

the

#

fucj

#

is file mapping suddenly broken

#

it's not even broken anywhere else

#

I think I see the bug

#

it's a use-after-free of physical memory

#

because I was freeing page cache memory used in a shared mapping

#

and on the second mapping of said file, it used that garbage data

#

which was caused by me using the wrong operator in the virtual mem function

#

and AHCI reading has decided to stop working

#

oh and psa to anyone reading this thread

#

the kernel has a mind of its own

#

I merely ask it to do stuff

flint idol Aug 27, 2024, 2:39 AM

#

flint idol and AHCI reading has decided to stop working

it seems to be a bug with the physical region populating code

#

as I thought

#

the physical regions cross page boundaries

#

causing the AHCI controller to write part of the data into some arbitrary region of memory

#

00000000395d3ea0-00000000395d40a0
000000003948a270-000000003948a470
000000003948a270-000000003948a470
000000003924df20-000000003924e120```

flint idol Aug 27, 2024, 3:08 AM

#

I fixed that

#

I also implemented --load-modules

#

to load kernel modules after PnP

#

anyway I'm going to go to sleep

#

TODO: Stop staying up late to do osdev

#

tomorrow, I'll make the probe drive for partition function thingy probe the partition for filesystems

#

I will also implement finalizing VFS init soon enough

#

(basically means mounting root)

flint idol Aug 27, 2024, 10:39 AM

#

flint idol I will also implement finalizing VFS init soon enough

this also does mean moving all special files under /dev into the new mount point

flint idol Aug 27, 2024, 11:23 AM

#

the fat driver can recognize partitions

flint idol Aug 27, 2024, 11:53 AM

#

#

#

after some slight shenangians

#

tomfoolery

#OBOS (not vibecoded)