lean glen Aug 16, 2024, 8:55 PM

#

would be simpler I think

flint idol Aug 16, 2024, 8:55 PM

#

the refcnt for the vnode includes dirents

#

so the page cache wouldn't be freed even if there aren't opened files

#

I'm going to add a mutex to the page cache entry

#

for locking IO on it

#

actually

#

I'll have two mutexes, one for modifying the structure (e.g., expanding the page cache size)

#

and another for the dirty region list

#

which I'll actually make an rb tree

#

maybe not

#

this is all just me thinking

#

not final

#

maybe not an rb tree

#

but regardless of the type of the structure

#

on read, the dirty list will be queried to see if there is a dirty entry

#

if so, it's locked

#

same thing on write

#

this way, reads on non-dirty regions can be done without any locking

#

and writes will ensure synchronization

#

(I finally spelled 'synchronization' right first try wow)

#

I'm also thinking that async io will be done using an event lock (not really a lock)

#

since it seems pretty good for the job

#

when the kernel is done the IO, it can set the event

#

causing any thread waiting on it to wake

#

and it can also be queried

#

so a thread can peridoically check if it's done

#

the work itself will probably be done on a worker thread

#

but the driver interface doesn't currently support async io

#

I'll do async io after fixing the page cache

flint idol Aug 16, 2024, 9:09 PM

#

flint idol same thing on write

when a write wants to add a dirty entry, it will first check if there is already one which contains the current offset

#

if so, it's expanded if needed

#

otherwise a new one is made

flint idol Aug 16, 2024, 9:28 PM

#

typedef LIST_HEAD(dirty_pc_list, struct pagecache_dirty_region) dirty_pc_list;
LIST_PROTOTYPE(dirty_pc_list, struct pagecache_dirty_region, node);
typedef struct pagecache_dirty_region
{
    // take this lock when any reads or writes are done on the region this represents.
    // also take it if expanding 'sz'
    mutex lock;
    size_t fileoff;
    size_t sz;
    LIST_NODE(dirty_pc_list, struct pagecache_dirty_region) node;
} pagecache_dirty_region;
typedef struct pagecache
{
    // Take this lock when expanding the page cache.
    mutex lock;
    char* data;
    size_t sz;
    // Take this lock when appending a node to the dirty region list.
    mutex dirty_list_lock;
    dirty_pc_list dirty_regions;
    atomic_size_t refcnt;
} pagecache;```

#

Those are my finalized page cache structs

flint idol Aug 16, 2024, 10:00 PM

#

now cached reads are as simple as:

pagecache_dirty_region* dirty = VfsH_PCDirtyRegionLookup(&desc->vn->pagecache, desc->offset);
if (dirty)
    Core_MutexAcquire(&dirty->lock);
if (desc->vn->pagecache.sz <= desc->offset)
    VfsH_PageCacheResize(&desc->vn->pagecache, desc->vn, desc->offset + nBytes);
memcpy(buf, desc->vn->pagecache.data + desc->offset, nBytes);
if (dirty)
    Core_MutexRelease(&dirty->lock);```

#

cached writes are essentially the same

#

except the memcpy goes the other way

#

page cache flushing is pretty easy:

driver_id* driver = vn->mount_point->fs_driver->driver;
for (pagecache_dirty_region* curr = LIST_GET_HEAD(dirty_pc_list, &pc->dirty_regions); curr; )
{   
    pagecache_dirty_region* next = LIST_GET_NEXT(dirty_pc_list, &pc->dirty_regions, curr);
    driver->header.ftable.write_sync(vn->desc, pc->data + curr->fileoff, curr->sz, curr->fileoff, nullptr);
    curr = next;
}```

#

I have no way to test writing atm, as I have no RW fs drivers

#

but I did implement it at least

#

however, it is practically a memcpy

#

for cached writes

#

damn

#

I have ||2244|| additions in two days

#

and ||98|| deletions

#

maybe I should start going outside

#

now time for async io

#

kernel is sitting at 26K loc

#

which is 3k more than my third kernel

#

this rewrite is starting to reach the state of the third kernel

#

all that's different is the fact that it had ~~userspace~~ code that ran in ring 3

#

I think async io will use a combination of event (signallable objects) and worker thread(s)

#

when an async read or write is made, the kernel starts a (kernel-mode?) thread

#

which does the IO, rather through the filesystem driver, or through the page cache, depending on the fd's flags

#

when the request is completed, the kernel will set the event

#

causing any thread waiting on it to wake

#

but if the data is already present in the page cache, no worker thread is made, and the operation completes immediately

#

(taking slight inspiration from NT)

flint idol Aug 16, 2024, 10:48 PM

#

flint idol but if the data is already present in the page cache, no worker thread is made, ...

the same thing happens if the data < the disk sector size

#

if the filesystem is not backed by a disk

#

then this case simply does not exist

flint idol Aug 16, 2024, 11:04 PM

#

flint idol but if the data is already present in the page cache, no worker thread is made, ...

however, if it is not already present, a worker thread is made which fills the missing parts

#

actually no

#

the data is copied over

#

but a worker thread is made to fill in the missing parts in the user buffer

#

and not the page cache

flint idol Aug 17, 2024, 1:01 AM

#

I might rewrite the initialization code soon

#

to not be so fucking cluttered

#

arch/x86_64/entry.c is 1100 lines

flint idol Aug 17, 2024, 1:24 AM

#

I am currently looking for a bug with the vma and huge pages

#

some funky shi be happenin'

#

currentNodeAddr < lastAddress

#

uintptr_t currentNodeAddr = currentNode->addr - (currentNode->addr % pgSize);

#

idk why I round it down

#

now it allocated the virtual memory

flint idol Aug 17, 2024, 1:32 AM

#

flint idol idk why I round it down

probably an artifact from the thing I ported it from

#

ok async reads seem to work properly...

#

I think I might use async reads while loading additional drivers from the cmdline

#

why? because async is cool

#

the vfs is probably nearly done

#

I just need to fix bugs

#

and also add some way to list directories

#

and open directories

#

but that won't be too much effort

#

and also unmount

#

that might take a bit

#

now I could do that tomorrow

#

or I could touch grass

inland radish Aug 17, 2024, 6:01 AM

#

tbh

#

consider trying this

MMPFN DirectPages[12], *IndirectPtr, **DoublyIndirectPtr, ***TriplyIndirectPtr /*etc*/;

main girder Aug 17, 2024, 6:15 AM

#

hey thats my idea.

inland radish Aug 17, 2024, 6:22 AM

#

main girder hey thats my idea.

yeah

#

it's also your idea

#

but it's also something ext2 does

main girder Aug 17, 2024, 6:23 AM

#

not just ext2

#

the very first unix filesystem did it

#

probably got the idea from multics

inland radish Aug 17, 2024, 6:27 AM

#

idk thats where I remember the idea from

flint idol Aug 17, 2024, 2:36 PM

#

inland radish consider trying this ``` MMPFN DirectPages[12], *IndirectPtr, **DoublyIndirectPt...

I don't quite understand the purpose of that

inland radish Aug 17, 2024, 2:37 PM

#

flint idol I don't quite understand the purpose of that

because it seems to me like you're allocating extra bytes and using extra time you dont need to use by keeping a linked list

flint idol Aug 17, 2024, 2:38 PM

#

That was an old pagecache entry

flint idol Aug 17, 2024, 2:38 PM

#

flint idol ```c typedef LIST_HEAD(dirty_pc_list, struct pagecache_dirty_region) dirty_pc_li...

This is a new one

flint idol Aug 17, 2024, 2:39 PM

#

inland radish because it seems to me like you're allocating extra bytes and using extra time y...

That is what I figured as well, which is why I revamped it before it was too late

flint idol Aug 17, 2024, 3:54 PM

#

I'm going to implement file mapping in a bit

#

shouldn't be too hard

#

just map some regions with the page cache

#

the parts that don't exist in the page cache

#

will be unmapped

#

so on page fault, that part of the page cache can be filled

#

and the pages can be mapped

lean glen Aug 17, 2024, 3:57 PM

#

Yeah then do CoW on those pages

flint idol Aug 17, 2024, 4:00 PM

#

flint idol I'm going to implement file mapping in a bit

*shared file mapping

#

for private maps, then yeah I'll do 🐄

vale nymph Aug 17, 2024, 4:16 PM

#

moo

inland radish Aug 17, 2024, 4:31 PM

#

flint idol I'm going to implement file mapping in a bit

what are you going to use to track file mappings?

#

I would use a tree keyed by initial address in your case

flint idol Aug 17, 2024, 4:31 PM

#

the normal page struct which is an rb-tree

inland radish Aug 17, 2024, 4:31 PM

#

and then use something like RB_NFIND except the condition is opposite to find the relevant mapping

flint idol Aug 17, 2024, 4:32 PM

#

so RB_FIND

inland radish Aug 17, 2024, 4:33 PM

#

RB_FIND finds an exact matching element

#

im talking about something that looks for the greatest element less than or equal to the key

#

RB_NFIND finds the least element greater than or equal to the key

flint idol Aug 17, 2024, 4:37 PM

#

so like:

page what = { ... };
page* found = RB_NFIND(page_tree, &context->pages, &what);
if (found->addr == what.addr) // the address is the key
  return found;
return RB_PREV(found);```

#

idk why I would want to do this instead of just finding the exact element

inland radish Aug 17, 2024, 4:47 PM

#

flint idol so like: ```c page what = { ... }; page* found = RB_NFIND(page_tree, &context->p...

that

inland radish Aug 17, 2024, 4:48 PM

#

flint idol idk why I would want to do this instead of just finding the exact element

cause you can fault on a page inside the region but not at the start

inland radish Aug 17, 2024, 4:48 PM

#

flint idol so like: ```c page what = { ... }; page* found = RB_NFIND(page_tree, &context->p...

this is what i am doing for another case

flint idol Aug 17, 2024, 4:57 PM

#

inland radish cause you can fault on a page inside the region but not at the start

my pf handler has code to round down the page
https://github.com/OBOS-dev/obos/blob/master/src/oboskrnl/arch/x86_64/entry.c#L254-L257

GitHub

obos/src/oboskrnl/arch/x86_64/entry.c at master · OBOS-dev/obos

I was bored... Contribute to OBOS-dev/obos development by creating an account on GitHub.

#

uintptr_t virt = getCR2();
virt &= ~0xfff;
if (Arch_GetPML2Entry(getCR3(), virt) & (1<<7))
    virt &= ~0x1fffff;```

#

maybe that'll be a problem for getting the offset

inland radish Aug 17, 2024, 5:04 PM

#

flint idol my pf handler has code to round down the page https://github.com/OBOS-dev/obos/b...

this only alleviates the problem within the page

#

certainly not within a range that spans several pages

#

eh you might be able to use the PTE to find the address of the VAD list entry

flint idol Aug 17, 2024, 8:07 PM

#

I am currently working on CoW

#

before implementing file mapping

#

it should work when the page is shared across two contexts

#

more than two contexts, likely not

#

actually maybe

#

I am also making OOM within page faults be handled better

#

if a page is being swapped in, but there is not enough physical memory, it will try to swap out enough pages to get enough

#

if it cannot

#

then it will block until there is enough memory

flint idol Aug 17, 2024, 8:19 PM

#

flint idol then it will block until there is enough memory

I think I will use a synchronization event to implement this

#

along with a linked list of threads, along with how much memory they are requesting

#

then whenever a page is swapped out, the VMM will check wake as many threads as it can

#

actually, I could just use a bare struct waitable_header to implement this

#

then keep how much memory the thread needs in struct thread

flint idol Aug 17, 2024, 8:22 PM

#

flint idol if a page is being swapped in, but there is not enough physical memory, it will ...

but this will only work if the pages are contiguous in phys. memory

#

so it'll only try to swap out one page

#

otherwise it will block until enough memory is freed

flint idol Aug 17, 2024, 8:28 PM

#

flint idol otherwise it will block until enough memory is freed

and to avoid code duplication in this case, the pmm will be made arch independant

#

it only took me 4 months to realize that I should've done that

flint idol Aug 17, 2024, 9:17 PM

#

the thingy broke

#

it's incorrectly calculating the physical memory boundaries somehow

#

this is all it is:

if ((phys + nPages * OBOS_PAGE_SIZE) > Mm_PhysicalMemoryBoundaries)
    Mm_PhysicalMemoryBoundaries = (phys + nPages * OBOS_PAGE_SIZE);```

#

at some point phys became garbage it seems

#

ok I fixed it

flint idol Aug 17, 2024, 10:51 PM

#

wait...

#

the pmm has been lock free

#

the entire duration of the kernel

#

and I have had no problems

chilly osprey Aug 17, 2024, 10:53 PM

#

clearly a sign that it can remain lockfree

flint idol Aug 17, 2024, 10:54 PM

#

I'll just keep it lock free until I have problems meme

elder hornet Aug 17, 2024, 10:55 PM

#

I have no idea what this means but it seems like nice progress :D

flint idol Aug 17, 2024, 10:55 PM

#

I think it is

#

I'll be honest idk what's happening here either

#

When I wrote this code, God and I knew what it did.

#

Now, only God knows.

lean glen Aug 17, 2024, 10:56 PM

#

chilly osprey clearly a sign that it can remain lockfree

meme

#

I'd lock at least the page queues

flint idol Aug 17, 2024, 10:57 PM

#

discord is being funny

chilly osprey Aug 17, 2024, 10:59 PM

#

i dont think that was a discord thing

flint idol Aug 17, 2024, 10:59 PM

#

why would he send the same message twice

lean glen Aug 17, 2024, 10:59 PM

#

I'm in a low cell coverage zone atm

flint idol Aug 17, 2024, 10:59 PM

#

but with a word swapped

lean glen Aug 17, 2024, 10:59 PM

#

I sent the first message then it disappeared somehow so I sent another one saying the same thing

flint idol Aug 17, 2024, 10:59 PM

#

ah ok

chilly osprey Aug 17, 2024, 11:01 PM

#

flint idol why would he send the same message twice

how would that even happen as a discord bug?

#

i could see it happening if they were identical

#

but swapping words seems unlikely

flint idol Aug 17, 2024, 11:01 PM

#

I thought he might've edited one of the messages

chilly osprey Aug 17, 2024, 11:01 PM

#

ah thats true yeah

flint idol Aug 17, 2024, 11:01 PM

#

but discord somehow sent it as a separate message

flint idol Aug 17, 2024, 11:17 PM

#

anyway, now that I have CoW implemented

#

I can implement file mapping probably

lean glen Aug 17, 2024, 11:19 PM

#

What method do you use for CoW?

#

Refcount then check on fault?

flint idol Aug 17, 2024, 11:20 PM

#

map pages as RO

#

then on fault, allocate a separate physical page

#

copy the contents there and remap one of the pages to use that phys. address instead

#

exit

#

this only works for two pages though

lean glen Aug 17, 2024, 11:22 PM

#

Ok but what if it's a page that wasn't forked

#

how do you know whether a page has been forked

flint idol Aug 17, 2024, 11:25 PM

#

lean glen how do you know whether a page has been forked

struct page* copied_page;              // If CoW is enabled on this page, this has a pointer to the page we're sharing data with.

#

basically if that's nullptr, the pf handler knows that the page faulted for some other reason

#

and not because of CoW

chilly osprey Aug 17, 2024, 11:38 PM

#

lean glen What method do you use for CoW?

i didnt know there were multiple methods

#

i was thinking you just

#

remap to a new page if it faults

#

and memcpy from the original

#

and set the write bit obviously

flint idol Aug 17, 2024, 11:38 PM

#

shit I almost forgot to memcpy

chilly osprey Aug 17, 2024, 11:39 PM

#

good thing i expressed my confusion

chilly osprey Aug 17, 2024, 11:44 PM

#

chilly osprey i didnt know there were multiple methods

nevermind im being stupid

#

i was thinking of a different situation which wouldnt really apply

flint idol Aug 17, 2024, 11:52 PM

#

I think I should maybe defer file mapping-related page faults

#

outside of the pf handler

#

and block until it's done

#

as I don't think reading from files and allocating memory for the page cache is a good idea while the pf handler is running

chilly osprey Aug 17, 2024, 11:59 PM

#

worker thread is always the solution

flint idol Aug 18, 2024, 12:01 AM

#

or I could implement APCs

#

then start a kernel APC on the current thread

#

that does the pf handler's work

#

or I could add:
TODO: Use kernel APCs instead
then use a DPC to do the work, then signal an event when it's done

#

which causes the thread to wake up

#

actually a kernel apc might not work

lean glen Aug 18, 2024, 12:08 AM

#

flint idol ```c struct page* copied_page; // If CoW is enabled on this page, t...

kinda sucks if the page is shared by multiple tasks

flint idol Aug 18, 2024, 12:08 AM

#

I know

lean glen Aug 18, 2024, 12:08 AM

#

chilly osprey i didnt know there were multiple methods

The mach way is to do object chaining and shadow pages and stuff

flint idol Aug 18, 2024, 12:08 AM

#

flint idol actually a kernel apc might not work

I could just wait until after the context's lock is released within the pf handler

lean glen Aug 18, 2024, 12:09 AM

#

I'm unsure where refcounting comes from, I took it from uvm but it may have been done in solaris

flint idol Aug 18, 2024, 12:09 AM

#

flint idol I could just wait until after the context's lock is released within the pf handl...

but there's the possibility that something else was locked when the pf happened causing a possible deadlock

flint idol Aug 18, 2024, 12:10 AM

#

flint idol or I could add: `TODO: Use kernel APCs instead` then use a DPC to do the work, t...

so I'll just use a DPC like I said here

chilly osprey Aug 18, 2024, 12:11 AM

#

lean glen The mach way is to do object chaining and shadow pages and stuff

that is all foreign to me

lean glen Aug 18, 2024, 12:11 AM

#

Yeah it's simpler and more efficient to do refcounting

chilly osprey Aug 18, 2024, 12:11 AM

#

refcounting is what i thought was the way to do it

#

yeah

lean glen Aug 18, 2024, 12:13 AM

#

Shadow objects implements copy on write. When a VM object is duplicated (e.g., at fork) a shadow object is created. A shadow object is initially empty, and points to underlying object. When the contents of a page of shadow object is modified, the page is copied and insert in the list of pages for the shadow object. A series of shadow objects pointing to shadow objects or original objects is a shadow chain.

#

https://pdos.csail.mit.edu/archive/6.097/lec/l18.html from this pdos MIT page

#

idk if this is a course, I know pdos is the MIT OS team tho

#

Looks cool anyway

flint idol Aug 18, 2024, 12:25 AM

#

flint idol so I'll just use a DPC like I said here

I might need to use a worker thread instead of a dpc

#

and I'm also going to need to start using mutexes more often

#

and converting some places to use mutexes instead of spinlocks

flint idol Aug 18, 2024, 12:27 AM

#

flint idol I might need to use a worker thread instead of a dpc

or I could just map in the page after the context lock is released

#

as I've thought about it, and unless someone does something schizophrenic, it'll be fine

chilly osprey Aug 18, 2024, 12:29 AM

#

lean glen https://pdos.csail.mit.edu/archive/6.097/lec/l18.html from this pdos MIT page

ill take a look at this, thanks

flint idol Aug 18, 2024, 12:34 AM

#

flint idol ```c struct page* copied_page; // If CoW is enabled on this page, t...

think I might change this to be a next and prev copied page pointer

#

so that it doesn't break with more than two pages doing CoW

#

I think this code is right if I want to insert a page inside a linked list:

if (pc_page->next_copied_page)
    pc_page->next_copied_page->prev_copied_page = page;
page->next_copied_page = pc_page->next_copied_page;
page->prev_copied_page = pc_page;
pc_page->next_copied_page = page;```

#

specifically to insert page in front of pc_page

lean glen Aug 18, 2024, 1:12 AM

#

chilly osprey ill take a look at this, thanks

Don't tho, just read the UVM paper for a refcount based approach

chilly osprey Aug 18, 2024, 1:14 AM

#

lean glen Don't tho, just read the UVM paper for a refcount based approach

i meant just to learn about it

#

i already intended on doing the refcounting

lean glen Aug 18, 2024, 1:15 AM

#

Yeah fair

chilly osprey Aug 18, 2024, 1:15 AM

#

it seems simple enough but ill read the paper anyway

lean glen Aug 18, 2024, 1:16 AM

#

You don't strictly need it but the concept of amaps is pretty simple and makes it easy to add other fancy features too

chilly osprey Aug 18, 2024, 1:17 AM

#

on failed write(page):
    page.refcount++;
    if page.refcount = 1:
        set write bit in page
    else:
        allocate new phys page and map it to virtaddr
        copy to the new page``` this is my current understanding basically

chilly osprey Aug 18, 2024, 1:17 AM

#

lean glen You don't strictly need it but the concept of amaps is pretty simple and makes i...

that is unfamiliar to me so i will definitely have a look

lean glen Aug 18, 2024, 1:18 AM

#

chilly osprey ``` on failed write(page): page.refcount++; if page.refcount = 1: ...

Yeah though it should be else if ref>1

#

and check perms too

chilly osprey Aug 18, 2024, 1:18 AM

#

lean glen Yeah though it should be else if ref>1

could it ever be zero?

#

i just incremented it above

lean glen Aug 18, 2024, 1:18 AM

#

yes

chilly osprey Aug 18, 2024, 1:18 AM

#

how?

lean glen Aug 18, 2024, 1:18 AM

#

no

#

nvm

#

Not in your case

chilly osprey Aug 18, 2024, 1:18 AM

#

ah

lean glen Aug 18, 2024, 1:19 AM

#

But it can and will drop to 0

#

then you can release it

chilly osprey Aug 18, 2024, 1:19 AM

#

but surely then it would be incremented again to 1 if another page writes to it

#

yeah that too

lean glen Aug 18, 2024, 1:20 AM

#

chilly osprey ``` on failed write(page): page.refcount++; if page.refcount = 1: ...

And refcount is not incremented per write but on copy

chilly osprey Aug 18, 2024, 1:20 AM

#

this is only on failed writes though which is the only time we would need to copy

#

or am i misunderstanding

lean glen Aug 18, 2024, 1:20 AM

#

refcount is used to keep track of how many times the page was copied

#

Well

#

Forked

chilly osprey Aug 18, 2024, 1:20 AM

#

yeah

#

ill just read the paper

#

it will save us both all of this back and forth

lean glen Aug 18, 2024, 1:21 AM

#

so you need to copy the page when it was written to and was forked (refcnt>1)

lean glen Aug 18, 2024, 1:21 AM

#

lean glen so you need to copy the page when it was written to and was forked (refcnt>1)

So you increase the refcnt in fork()

ornate ginkgo Aug 18, 2024, 7:07 AM

#

chilly osprey it seems simple enough but ill read the paper anyway

just chiming in to say I also recommend the uvm paper, it's got some (imo) really interesting thoughts in it. None of which I've applied yet nooo

flint idol Aug 18, 2024, 11:37 AM

#

something weird happened

#

added:

ctx->workingSet.size -= (page->prot.huge_page ? OBOS_HUGE_PAGE_SIZE : OBOS_PAGE_SIZE);

#

when a page is removed from the working set

#

to be greeted with:

ASAN Violation at ffffffff80053ea9 while trying to write 8 bytes from 0xffff80000257f0b8 (Hint: Use of memory block after free).```

#

uStressTester strikes once again nooo

#

(that's what I'm calling uACPI from now on)

main girder Aug 18, 2024, 11:50 AM

#

have you not stress tested your heap

flint idol Aug 18, 2024, 11:51 AM

#

I have

#

although this seems to be a bug in my vmm

#

that wasn't caught

#

because of another bug in my vmm

main girder Aug 18, 2024, 11:53 AM

#

describe it

flint idol Aug 18, 2024, 11:55 AM

#

The vmm swaps out pages (specifically those in the working-set that are getting evicted) in the page replacement algo code. I seem to have forgotten to subtract that page from the size of the working-set pages (context->workingSet.size)

#

causing the working-set to keep the same pages in it practically forever

#

the fix to that bug uncovered other bugs in the vmm

main girder Aug 18, 2024, 11:57 AM

#

like ?

flint idol Aug 18, 2024, 11:58 AM

#

use after frees because nodes aren't being removed from structures

main girder Aug 18, 2024, 12:00 PM

#

ur swapping is bad btw

flint idol Aug 18, 2024, 12:02 PM

#

how come?

main girder Aug 18, 2024, 12:03 PM

#

you do the same naive thing i did at first where you write out 1 (one) page and then immediately free it

#

probably on demand in response to a memory allocation that would otherwise fail

#

MmH_FindAvaliableAddress

#

typo

#

avaliable

flint idol Aug 18, 2024, 12:07 PM

#

what the H

#

?

#

oh

real pecan Aug 18, 2024, 12:08 PM

#

ava liable

real pecan Aug 18, 2024, 12:08 PM

#

main girder you do the same naive thing i did at first where you write out 1 (one) page and ...

whats the non naive approach

main girder Aug 18, 2024, 12:08 PM

#

what thehell is this doing...

flint idol Aug 18, 2024, 12:09 PM

#

supposed to initialize a spinload

#

*spinlock

#

shit

#

spinlocks aren't just atomic flags anymore

main girder Aug 18, 2024, 12:09 PM

#

real pecan whats the non naive approach

in an NT style vmm its

#

when you trim a page from a working set (or its refcount otherwise hits 0 but thats the most common reason)

#

and it is marked dirty

#

i.e., it has either never been written to the pagefile, or, it has been written to since the last time it was written to the pagefile

#

you place it on the 'modified page list'

#

when the number of pages on that list exceeds a threshold OR memory gets low, a thread called the modified page writer is awoken to 'clean' those pages by allocating contiguous clusters in the pagefile and writing them out in as big of clustered IOs as you can

#

this does not reclaim those pages immediately

#

they are merely marked clean and placed on the 'standby list'

#

which is where clean (non-dirty) pages go when their refcount hits 0

#

page frames on the standby list are considered free memory

#

and when the free list runs out, you yoink a standby page

#

for instance most of the page cache at any given time will (probably) be on the standby list

#

when both lists run out, by that point the modified page writer has been signaled to start writing pages for one reason or another and some will be available on the standby list at some point

#

so you block until they are

#

if you were holding any locks that need to be grabbed in order to free memory (such as your working set lock) then you drop them before blocking for free pages otherwise youll cause a deadlock

#

which may require you to back out of whatever half-finished thing you were doing and then retry it after youre unblocked and can reacquire the lock

main girder Aug 18, 2024, 12:17 PM

#

main girder when the number of pages on that list exceeds a threshold OR memory gets low, a ...

file cache pages can also go on the modified list in which case they are written back to the appropriate file

#

everything i just said applies to anonymous (pagefile-backed) pages but analogous things occur with file pages as well

main girder Aug 18, 2024, 12:18 PM

#

main girder page frames on the standby list are considered free memory

also an important detail here is that you place pages on the back of the standby list so that a more frequently used page is less likely to get reclaimed soon

#

which means you have more time, if you happen to need it again, to fault it back into your working set before it gets reclaimed

#

and then if it gets trimmed again and you only read from it while it was mapped, itll go back on the standby list immediately when trimmed cuz it already has valid backing in the pagefile from last time it was trimmed and went on the modified list

#

but if you wrote to it then youve dirtied it which invalidates its backing in the pagefile

#

so then itd go back on the modified page list

flint idol Aug 18, 2024, 12:29 PM

#

flint idol although this seems to be a bug in my vmm

it was because of how I would detect if a page was in the referenced list

#

I would check if age == 1

#

but once the sampling interval ends, age is shifted left by one

#

but sometimes, I guess the page isn't removed from the referenced list at the end of the sampling interval

#

now a garbage physical address is being passed somewhere

#

:/

#

I added an assertion to allocate physical pages:

Assertion failed in function Mm_AllocatePhysicalPages. File: /home/oberrow/Code/obos/src/oboskrnl/mm/pmm.c, line 142. phys < Mm_PhysicalMemoryBoundaries```

#

maybe I should add a lock to the pmm

#

that didn't fix the issue (at hand)

flint idol Aug 18, 2024, 1:07 PM

#

I was passing a byte count instead of a page count to Mm_FreePhysicalPages in one specific case

#

but otherwise, I get

Stack corruption detected at IP=0xffffffff80005788 (overwrite of stack canary).```

#

"fixed" that

#

because now I get

Page fault at 0xffffffff8000516a in kernel-mode while to write page at 0x0000000000000000, which is unpresent. Error code: 2```

#

page* node = Mm_Allocator->ZeroAllocate(Mm_Allocator, 1, sizeof(page), &status);

#

that's where it faulted

#

nooo

#

~~bug doesn't happen on m68k, x86_64 support is dropped~~

#

the m68k port has so much less bullshit going in it

#

[ DEBUG ] Arch_KernelEntry: Initializing allocator.
[ DEBUG ] Arch_KernelEntry: Initialize kernel process.
[ DEBUG ] Arch_KernelEntry: Initializing IRQ interface.
[ DEBUG ] Arch_KernelEntry: Initializing VMM.
[ DEBUG ] Arch_KernelEntry: Initializing timer interface.
[ DEBUG ] Arch_KernelEntry: Initializing scheduler timer.
[ DEBUG ] Arch_KernelEntry: Loading kernel symbol table.
[ DEBUG ] Arch_KernelEntry: Loading test driver.
[ DEBUG ] Drv_LoadDriver: Loaded driver at 0xc04a3378.```
this is the log on the m68k

#

the one for x86_64 doesn't fit in a message

#

all for the exact same functionality (except for PnP)

flint idol Aug 18, 2024, 1:48 PM

#

don't we just love it when an allocation of 13 bytes fails

#

it was because huge pages aren't supported on the m68k (at least I don't know of such thing for the m68040's mmu)

#

causing the map function to return unimplemented

#

after fixing that I can do all x86_64 can

#

but with about 30 less files

#

and it doesn't crash either

#

no way I actually got the x86-64 port to OOM

ashen goblet Aug 18, 2024, 2:03 PM

#

has the obos-curse returned?

flint idol Aug 18, 2024, 2:03 PM

#

only for x86-64

ashen goblet Aug 18, 2024, 2:04 PM

#

maybe it's just hidden on m68k but still there

flint idol Aug 18, 2024, 2:04 PM

#

actually this isn't OOMing, I'm simply running out of swap space

#

one would think 64 mib would be enough

lean glen Aug 18, 2024, 2:05 PM

#

now compress it

flint idol Aug 18, 2024, 2:11 PM

#

nvm, the in-ram swap is getting deep fried

#

fixed that

#

but the swap still claims to have no space

#

(there's only ~15000 nodes with the exact amount of space needed)

#

I fixed that

#

to get a ubsan violation in swap_free

flint idol Aug 18, 2024, 3:39 PM

#

my in-ram temporary swap seems to be very broken

#

it's causing everything from ubsan violations to page faults to KASAN violations

#

bro wtf

#

when I copy+paste the same thing, but on the m68k port

#

it's completely fine

#

like the initial swap thing was copy+pasted from x86_64's implementation

#

all that's different are the IRQLs being raised

flint idol Aug 18, 2024, 5:00 PM

#

why is this shit like this

#

the m68k driver loader has decided to break

flint idol Aug 18, 2024, 7:04 PM

#

flint idol the m68k driver loader has decided to break

I wonder if this has anything to do with the fact I use my .pageable.* sections

#

it saw

#

*was

#

the ld script is broken

inland radish Aug 18, 2024, 7:24 PM

#

flint idol I wonder if this has anything to do with the fact I use my `.pageable.*` section...

i'd honestly make them part of .text

#

like .text.init, .text.page

flint idol Aug 18, 2024, 7:24 PM

#

won't work, I have to be able to separate them through PHDRs

#

it's not quite allowed to use sections within an elf (program) loader

inland radish Aug 18, 2024, 7:28 PM

#

flint idol won't work, I have to be able to separate them through PHDRs

why

inland radish Aug 18, 2024, 7:28 PM

#

flint idol it's not quite allowed to use sections within an elf (program) loader

no

#

you dont need to do that

#

you simply do like

#

1s

#

    /* Define the regular text section. */
    .text : {
        *(.text)
    } :text
    . = ALIGN(CONSTANT(MAXPAGESIZE));
    
    /* Define the initialization text section, which will be permanently reclaimed after system initialization. */
    KiTextInitStart = .;
    .text.init : {
        *(.text.init)
    } :text
    . = ALIGN(CONSTANT(MAXPAGESIZE));
    KiTextInitEnd   = .;
    
    /* Define the paged text section, which may not be resident at all times, pages may be swapped out and
       back in at the memory manager's discretion.  */
    KiTextPageStart = .;
    .text.page : {
        *(.text.page)
    } :text
    . = ALIGN(CONSTANT(MAXPAGESIZE));
    KiTextPageEnd   = .;
    
    /* Glob all of the other text sections into .text. */
    .text : {
        *(.text.*)
    } : text

#

this is what I do in boron

#

they all belong in the text phdr

#

the sections themselves are aligned so that they dont straddle page boundaries which would make pageout/reclamation impossible because part of untouched .text is in the way

flint idol Aug 18, 2024, 7:39 PM

#

time to lock in and get file mapping done

#

I've done like everything for file mapping- except for file mapping itself (i.e., in the virtual memory allocate function)

flint idol Aug 18, 2024, 8:33 PM

#

obos_status status = Vfs_FdOpen(&file, pathspec, FD_OFLAGS_READ_ONLY);
if (obos_is_error(status))
{
    OBOS_Debug("Could not open %s. Status: %d\n", pathspec, status);
    goto end;
}
Vfs_FdSeek(&file, 0, SEEK_END);
size_t filesize = Vfs_FdTellOff(&file);
Vfs_FdSeek(&file, 0, SEEK_SET);
char *buf = Mm_VirtualMemoryAlloc(&Mm_KernelContext, NULL, filesize, 0, VMA_FLAGS_HUGE_PAGE, &file, nullptr);
OBOS_Debug("Mapped %s. Contents:\n%*s\n", pathspec, filesize, buf);```

#

file mapping works

#

at least shared mapping does...

#

if the region is already in the page cache, it crashes

#

oops

#

now the region can already be in the page cache

#

as I expected, private mapping doesn't work

#

I think that's fixed

#

except I don't do CoW

#

for private mappings

#

I'll do that "quickly"

#

CoW seems to work

#

maybe that's mainly because there are no writes...

#

CoW works!

#

although I think there could be a bug where a page originally marked as RO can become RW if it's marked as CoW

#

before I commit this, I just need to make one last change with shared mappings

#

so that they are marked as dirty in the page cache if they are modified

flint idol Aug 18, 2024, 11:08 PM

#

flint idol so that they are marked as dirty in the page cache if they are modified

and to do this, I think I should be able to just mark the page as RO, then on fault, add the region to the dirty list, then mark the page RW, then return

flint idol Aug 19, 2024, 12:13 AM

#

I think shared and private file maps work

#

just pushed the code

vale nymph Aug 19, 2024, 12:20 AM

#

syscalls/mlibc now?

flint idol Aug 19, 2024, 12:20 AM

#

TODOs for tomorrow:

Directory handles
Making sure devices work when mapped on a vnode
Maybe unmounting

flint idol Aug 19, 2024, 12:20 AM

#

vale nymph syscalls/mlibc now?

probably in a couple days

#

I want to refactor the x86-64 code, as it is quite a mess

flint idol Aug 19, 2024, 12:21 AM

#

flint idol TODOs for tomorrow: - Directory handles - Making sure devices work when mapped o...

might as well do the 2nd right now

#

mapping the uart driver's file works flawlessly on the first try

#

idk about freeing though

#

that might be a problem

#

Oh my god I might be dumb

#

the virtual memory free never releases any physical pages

ornate ginkgo Aug 19, 2024, 12:29 AM

#

lol

vale nymph Aug 19, 2024, 12:30 AM

#

who needs to release memory anyways computers have millions of pages

ornate ginkgo Aug 19, 2024, 12:30 AM

#

just put everything important in the swap space, reset the machine, memory freed!

flint idol Aug 19, 2024, 12:31 AM

#

so if the file isn't a mapped file or it's a shared mapping, then I think it's safe to free the physical page

#

but now something decides to hang

#

or maybe the PMM is just really, really slow

#

nvm something's hanging

#

I hate tlb shootdowns

#

this is the millionth time something's hung because one cpu decided it was special and didn't want to handle a tlb shootdown

#

idk how safe it is to wait until after the context lock is released to unmap pages

#

might cause a race condition in which one thread tries to map the reclaimed pages while they're being freed

main girder Aug 19, 2024, 12:50 AM

#

context lock?

flint idol Aug 19, 2024, 12:51 AM

#

a lock for a vmm context

#

https://github.com/OBOS-dev/obos/blob/32c4d4977f26f4827d52b2fea733d523dd7d3fdf/src/oboskrnl/mm/context.h#L60

GitHub

obos/src/oboskrnl/mm/context.h at 32c4d4977f26f4827d52b2fea733d523d...

I was bored... Contribute to OBOS-dev/obos development by creating an account on GitHub.

flint idol Aug 19, 2024, 1:35 AM

#

I can write to a serial port using the VFS!

#

I now need to fix a bug with the UART driver though

#

basically, a synchronous read is asynchronous

#

pathspec = "/dev/COM1";
Vfs_FdOpen(&file, pathspec, FD_OFLAGS_UNCACHED);
Vfs_FdIoctl(&file, 6, /* IOCTL_OPEN_SERIAL_CONNECTION */0, 
    1,
    115200,
    /* EIGHT_DATABITS */ 3,
    /* ONE_STOPBIT */ 0,
    /* PARITYBIT_NONE */ 0,
    &file.vn->desc);
file.vn->un.device->desc = file.vn->desc;
char message[15] = {};
Vfs_FdRead(&file, message, 14, nullptr);
Vfs_FdWrite(&file, message, 14, nullptr);```

#

If you write 14 bytes onto the serial port, it will echo them back to you

#

(most exciting 14 bytes I've had in a while)

#

tomorrow, I will work on async IO from devices

#

as it needs to be handled slightly differently to async IO on files

#

actually maybe not

flint idol Aug 19, 2024, 1:54 AM

#

flint idol tomorrow, I will work on async IO from devices

*pipe-style devices (such as a UART)

#

I will also add an API to the driver interface which adds a special file to /dev

flint idol Aug 19, 2024, 11:27 AM

#

flint idol tomorrow, I will work on async IO from devices

I have thought about this, and there isn't really much I need to do

flint idol Aug 19, 2024, 1:19 PM

#

I've added some APIs to allocate vnodes and dirents for drivers

#

I am now working on block devices to make sure they work properly

flint idol Aug 19, 2024, 1:58 PM

#

flint idol TODOs for tomorrow: - Directory handles - Making sure devices work when mapped o...

Directory handles are unneeded, as I can just use dirents (for now)

#

in fact maybe even forever

#

time to implement unmounting nooo

#

in theory, this will be simple

#

for each dirent, deref the vnode (decrement its refcount, and free it once the ref count reaches zero)

#

then free the current dirent

#

then I will also need to close any open files on the mount point, and flush each vnode's page cache

flint idol Aug 19, 2024, 2:10 PM

#

flint idol then I will also need to close any open files on the mount point, and flush each...

and also free the namecache

#

and wait for any pending async IO operations to finish

#

Now in order, the unmount function needs to:

Lock the mount point
Close all opened files.
Wait for pending async IO to finish
Free the name cache.
Flush the page cache
Dereference each vnode within the mount point
Free each dirent
Unlock the mount point, and return

flint idol Aug 19, 2024, 4:30 PM

#

theoretically, I should be able to unmount now...

flint idol Aug 19, 2024, 4:48 PM

#

I can't because my foreach_dirent doesn't work

#

mainly because skill issue

#

(idk how to recursively iterate over the tree without using recursion)

inland radish Aug 19, 2024, 4:57 PM

#

flint idol (idk how to recursively iterate over the tree without using recursion)

you can

#

simply fetch the "successor" node each time

#

assuming a binary tree

flint idol Aug 19, 2024, 4:57 PM

#

it's not a binary tree

inland radish Aug 19, 2024, 4:57 PM

#

which kind of tree is it

flint idol Aug 19, 2024, 4:57 PM

#

the kind where a node can have any amount of children

inland radish Aug 19, 2024, 4:58 PM

#

so why are you storing dirents

#

is it some kind of cache?

flint idol Aug 19, 2024, 4:58 PM

#

no it's for directory/file lookup

inland radish Aug 19, 2024, 4:59 PM

#

oh path lookup

flint idol Aug 19, 2024, 4:59 PM

#

yeah

inland radish Aug 19, 2024, 5:00 PM

#

so whats path lookup have to do with unmounting

flint idol Aug 19, 2024, 5:01 PM

#

inland radish so whats path lookup have to do with unmounting

each mount point contains the dirent tree for path lookup

#

and I gotta free it on unmount

inland radish Aug 19, 2024, 5:02 PM

#

you can store all the dirent nodes on that mountpoint in a list as well as the path tree

flint idol Aug 19, 2024, 5:02 PM

#

that's what I'm thinking to do instead

inland radish Aug 19, 2024, 5:02 PM

#

you can do both

#

have a path tree to assist lookup, AND store them in a list to assist their deallocation on unmount

flint idol Aug 19, 2024, 5:03 PM

#

ye

flint idol Aug 19, 2024, 5:54 PM

#

I have finished implementing unmounting

#

now, all I have to do is make all this thread-safe

white mulch Aug 19, 2024, 6:44 PM

#

flint idol the kind where a node can have any amount of children

vector<Node*> stack {root};
while (!stack.is_empty()) {
    Node* node = stack.pop();

    // do whatever with node

    for (Node* child : node->children) {
        stack.push(child);
    }
}
``` assuming you are fine with top down depth first (as in if you eg. have a subtree with children then the childs are visited before the subtree's siblings

flint idol Aug 19, 2024, 7:08 PM

#

accidentally got off track and yapped to someone about multithreading (oops)

#

and synchronization things

#

but now I'll make this thread-safe

flint idol Aug 19, 2024, 7:38 PM

#

VFS unmounting is done

#

this should all be kind of thread safe

#

almost™ time for userspace

flint idol Aug 19, 2024, 7:55 PM

#

for future reference:
if you want to iterate over a directory, you call VfsH_DirentLookup to get the directory's dirent, then you set your curr node to the first child of it (dir->d_children.head), then you iterate over that like it were it a linked list (curr = curr->next)

#

that's just so I can look back though if I forget how

flint idol Aug 19, 2024, 8:11 PM

#

so I think I can merge the VFS

#

now

flint idol Aug 19, 2024, 9:33 PM

#

flint idol so I think I can merge the VFS

probably

#

Finally, no merge conflicts

#

oh god

#

in 4 days too

#

I think I just saw some green stuff on the floor outside- what is that?

#

along with 3225 LoC added total

#

https://github.com/OBOS-dev/obos/pull/20

GitHub

Vfs by oberrow · Pull Request #20 · OBOS-dev/obos

VFS Code + InitRD, alongside some miscellaneous bug fixes.

flint idol Aug 19, 2024, 10:01 PM

#

I can almost start porting stuff

#

I just need to implement a couple things

#

Anyone knows what this is

#

?

elder hornet Aug 19, 2024, 10:15 PM

#

whats that

#

must be a glitchy framebuffer test

flint idol Aug 19, 2024, 10:16 PM

#

I went outside of the door of my house

#

And found it

#

I think I might be lagging

#

@real pecan

#

Do you know what that is

real pecan Aug 19, 2024, 10:17 PM

#

id need a spec to tell u

elder hornet Aug 19, 2024, 10:17 PM

#

flint idol I went outside of the door of my house

YOU CAN EXIT HOUSES????

flint idol Aug 19, 2024, 10:18 PM

#

elder hornet YOU CAN EXIT HOUSES????

I was surprised too

#

In fact the air out here smells different

#

Idk whether I like it or not

elder hornet Aug 19, 2024, 10:18 PM

#

must be poisonous. Outside is dangerous

flint idol Aug 19, 2024, 10:19 PM

#

There's even some great big light

#

That's hurting my eyes

elder hornet Aug 19, 2024, 10:21 PM

#

Yeah just sounds like a glitchy framebuffer to me

#

Do you know who manufactured it? Maybe you can get a refund

flint idol Aug 19, 2024, 10:23 PM

#

ok I'm back in

flint idol Aug 19, 2024, 10:24 PM

#

flint idol I just need to implement a couple things

those things being:

pnp with drivers from the initrd
AHCI and FAT drivers
finalizing VFS initialization

#

then I just implement a bunch of syscalls

#

then I port mlibc

#

then I suffer

flint idol Aug 19, 2024, 10:25 PM

#

flint idol those things being: - pnp with drivers from the initrd - AHCI and FAT drivers - ...

basically what finalizing vfs initialization does is mounts the proper root fs

#

instead of the initrd

#

and then eventually I'll write documentation

#

eventually

eventually
-# eventually

flint idol Aug 19, 2024, 11:07 PM

#

flint idol those things being: - pnp with drivers from the initrd - AHCI and FAT drivers - ...

time to do some PnP things

flint idol Aug 19, 2024, 11:43 PM

#

I can now throw drivers into the InitRD and have them detected and laoded

#

*loaded

vernal chasm Aug 19, 2024, 11:47 PM

#

O_O

#

PogW

#

Idk what I'm responding to, but I felt obligated to respond with that kekw

flint idol Aug 19, 2024, 11:47 PM

#

#

basically the kernel can detect devices

vernal chasm Aug 19, 2024, 11:47 PM

#

PogW

flint idol Aug 19, 2024, 11:47 PM

#

then find the right driver for it

#

if one exists

vernal chasm Aug 19, 2024, 11:47 PM

#

So you added that back in now

flint idol Aug 19, 2024, 11:47 PM

#

Yes

vernal chasm Aug 19, 2024, 11:48 PM

#

Now make it possible to get validated drivers from the internet

flint idol Aug 19, 2024, 11:48 PM

#

KEKW

#

I also need to load drivers as passed on the kernel cmd line

#

--load-modules=driver,driver,...

#

#

this should even work on real hw

#

Forgot uacpi broke stiff

#

*stuff

#

But I'll need to fix that for pnp to work

#

Just noticed there is a security vulnerability in OBOS

#

Basically if you free a vnode which includes the page cache

#

You get a use after free in any virtual memory region that's allocated it

#

But specifically in the physical pages backing the pages, which is undetectable by the kernel

#

Which means someone (no one) could exploit this to get access to arbitrary data

#

Wouldn't say the severity is too high though, since the data is undefined

#

unless...

#

it uses VfsH_PageCacheResize with the size parameter set to zero to free the page cache

#

and this function does seem to remap the mapped regions

flint idol Aug 20, 2024, 12:13 AM

#

flint idol You get a use after free in any virtual memory region that's allocated it

*any shared virtual memory region mapping said vnode

#

although there is still a use-after-free

#

but it's not related to the mapped regions themselves

#

instead it's related to the struct pages

flint idol Aug 20, 2024, 2:21 AM

#

@real pecan I think your thing might actually be broken this time. I decided to run the uACPI test runner on my computers DSDT and SSDTs to see if uACPI was failing, and to my surprise, something did seem to be broken

#

all goes well up to the point where my kernel crashes

#

uACPI's test runner's last message is:

[uACPI][ERROR] failed to disable fixed event 0```

#

before exiting

#

I'll set uACPI to trace

#

and see what happens

#

I can also try to run OBOS on other hardware

#

to see if it still crashes there

#

I can also hopefully run managarm to see if it crashes aswell

#

currently trying to build managarm

flint idol Aug 20, 2024, 2:51 AM

#

xbstrap: Action regenerate of source mlibc failed```

#

the managarm isn't managarming

#

I think I got that working

#

this is so slow

#

dam

#

there was a nightly build this entire time

flint idol Aug 20, 2024, 3:44 AM

#

Can't be bothered with that any more

#

Only thing I can think of tbh is a vmm bug assuming it's my problem

#

I'll try on another piece of hw

flint idol Aug 20, 2024, 3:45 AM

#

flint idol Only thing I can think of tbh is a vmm bug assuming it's my problem

Because it crashes

#

Then crashes while crashing

#

Then just goes on recursively

flint idol Aug 20, 2024, 4:17 AM

#

Forgot that hyper had broken on the other test subject

flint idol Aug 20, 2024, 4:37 AM

#

On the other test subject it fails

#

Likely do to make kernel api

#

As it said invalid srgument

#

*argument

real pecan Aug 20, 2024, 6:47 AM

#

flint idol uACPI's test runner's last message is: ``` [uACPI][ERROR] failed to disable fixe...

Thats normal

#

It cant perform io properly in userspace, so whatever it wrote it didn't read back

#

Thats a to-do for later

#

But thats not a bug nor a problem

#

Just extra log spam

#

Btw the runner is compiled with asan and ubsan, and also aborts on leaks, so if it did do something wrong you would see it

flint idol Aug 20, 2024, 9:41 AM

#

Ah ok

#

Also when hyper old uefi bug fix

real pecan Aug 20, 2024, 10:00 AM

#

Soon ™️

elder hornet Aug 20, 2024, 11:41 AM

#

Idk why but I find your VFS monologues very interesting

flint idol Aug 20, 2024, 2:24 PM

#

Good to know

flint idol Aug 20, 2024, 2:57 PM

#

I was able to get uacpi to initialize on real hw when I disabled the gsi

real pecan Aug 20, 2024, 3:02 PM

#

Which gsi

flint idol Aug 20, 2024, 3:03 PM

#

#9 for GPEs

real pecan Aug 20, 2024, 3:03 PM

#

Skill issue

flint idol Aug 20, 2024, 3:03 PM

#

I am aware

real pecan Aug 20, 2024, 3:03 PM

#

Lol

flint idol Aug 20, 2024, 3:04 PM

#

why it causes the kernel to crash, I don't know

#

and why it causes the kernel to be in an infinite panic loop, I don't know

flint idol Aug 20, 2024, 3:40 PM

#

There is 7676 kib of pageable pages on this hw

#

And 9328 kib paged out

#

Time to start an AHCI driver

lean glen Aug 20, 2024, 3:41 PM

#

if u wanna port stuff right now you could too

flint idol Aug 20, 2024, 3:41 PM

#

I know

lean glen Aug 20, 2024, 3:41 PM

#

use a tmpfs

#

it'd be easier to test on real hardware too

flint idol Aug 20, 2024, 3:42 PM

#

I want to write this first

#

Before ports

#

I also need a syscall interface before ports

flint idol Aug 20, 2024, 4:18 PM

#

finally found a use for semaphores in the kernel

#

I can use them in the AHCI driver for ports

#

since you have a max of 32 command slots

vale nymph Aug 20, 2024, 4:23 PM

#

Thats one of the places where I use them in astral too, but for nvme queues and stuff

flint idol Aug 20, 2024, 5:16 PM

#

the text renderer thing has decided to break on me

#

in flush_buffers

flint idol Aug 20, 2024, 5:54 PM

#

bro wtf

#

when I comment out everything in get_bar_size in the pci file

#

all goes back to normal

flint idol Aug 20, 2024, 6:31 PM

#

I got very little code in the AHCI driver, but it's a start

#

just detects the AHCI controller after being loaded by pnp

#

then maps the HBA

#

I'll do more in a bit

flint idol Aug 20, 2024, 6:34 PM

#

flint idol I got very little code in the AHCI driver, but it's a start

Weirdly the ahci controller is in the same place on real hw, not that it's a problem or anything

#

Anyway, I need to initialize the hba

#

Then detect and initialize the ports

#

Then send identify ata to them

#

Then I need to implement reading and writing from the ports

flint idol Aug 20, 2024, 6:36 PM

#

flint idol Then send identify ata to them

Before this I need to get an irq handler for this

#

Then that's probably all I'm going to do for the ahci driver for a long time

#

As there isn't much I want to do with it

flint idol Aug 20, 2024, 6:37 PM

#

flint idol Then send identify ata to them

For each port I do this on, I register a vnode

#

For said port

vernal chasm Aug 20, 2024, 6:49 PM

#

When do we get obos graphics?

flint idol Aug 20, 2024, 7:02 PM

#

Eventually

flint idol Aug 20, 2024, 8:34 PM

#

some funny stuff is happening

#

basically something is remapping already-mapped pages

#

while it doesn't seem to be a problem directly

#

it's causing the kernel to hang waiting for a tlb shootdown to finish

#

one cpu has decided it's special

#

and doesn't want to handle the tlb shootdown irq

#

because eflags.if=false

#

because it's in a pf handler

#

which is a bug

#

which is easily solved by sti

vernal chasm Aug 20, 2024, 9:26 PM

#

flint idol basically something is remapping already-mapped pages

O_O

#

Good job

flint idol Aug 20, 2024, 10:54 PM

#

vernal chasm O_O

haesn't caused a problem

#

something could've just been examining the state wrong

vernal chasm Aug 20, 2024, 11:02 PM

#

kekw

flint idol Aug 20, 2024, 11:03 PM

#

-# since it doesn't crash, it's fine

flint idol Aug 21, 2024, 12:46 AM

#

I've had:

 223 files changed, 25330 insertions(+), 866 deletions(-)```
this summer

#

along with 20k LoC added

#

along with 6 PRs merged

flint idol Aug 21, 2024, 1:12 AM

#

flint idol Weirdly the ahci controller is in the same place on real hw, not that it's a pro...

Even on another test subject

#

What is so special about slot 31, function 2

#

On bus zero

ornate ginkgo Aug 21, 2024, 5:57 AM

#

flint idol What is so special about slot 31, function 2

from what i've seen bus 0, slot 31 is where the chipset likes to live.

#

so you get lpc, ahci, smbus and some other peripherals there

ornate ginkgo Aug 21, 2024, 6:02 AM

#

ornate ginkgo from what i've seen bus 0, slot 31 is where the chipset likes to live.

actually im not sure about that, that was based on a few intel systems I'd seen, but the one I'm on currently is amd and has the chipset on bus 1, slot 0.

thick jolt Aug 21, 2024, 9:32 AM

#

ping?

flint idol Aug 21, 2024, 12:17 PM

#

Pong

flint idol Aug 21, 2024, 12:18 PM

#

ornate ginkgo actually im not sure about that, that was based on a few intel systems I'd seen,...

Guess that explains it since both machines use intel cpus

flint idol Aug 21, 2024, 1:39 PM

#

I have just added a way to allocate physical memory but using strictly 32-bit addresses

#

will be useful for supporting old ahci controllers that don't have 32-bit addressing

#

and other devices

inland radish Aug 21, 2024, 3:11 PM

#

flint idol will be useful for supporting old ahci controllers that don't have 32-bit addres...

*don't have 64-bit addressing

flint idol Aug 21, 2024, 3:11 PM

#

yeah that's what I meant, mb

#

#

I can initialize the HBA, as well as the ports

#

I just need to send identify ata to get drive info

flint idol Aug 21, 2024, 5:27 PM

#

the coolest thing about this rewrite is that it will turn on the drive led

#

in the ahci driver

vale nymph Aug 21, 2024, 5:36 PM

#

flint idol the coolest thing about this rewrite is that it will turn on the drive led

Holy shit obos best os

flint idol Aug 21, 2024, 5:36 PM

#

ikr

flint idol Aug 21, 2024, 5:36 PM

#

flint idol the coolest thing about this rewrite is that it will turn on the drive led

nothing that I've done over the past 3 or 4 months matters, drive led is highest priority

inland radish Aug 21, 2024, 5:42 PM

#

flint idol the coolest thing about this rewrite is that it will turn on the drive led

Oh it's controlled manually?

flint idol Aug 21, 2024, 5:43 PM

#

I could've sworn it was...

#

I was looking into the ahci spec to see what I needed to do

#

but found nothing

inland radish Aug 21, 2024, 5:44 PM

#

That's so...

#

...stupid!

flint idol Aug 21, 2024, 5:44 PM

#

I'm testing to see if commands are sent

#

... and Core_SemaphoreAcquire is a hidden symbol

#

Kernel Panic on CPU 2 in thread 4, owned by process 1. Reason: OBOS_PANIC_FATAL_ERROR. Information on the crash is below:
Unhandled NMI!```

#

thonk

#

while sending an IPI

haughty abyss Aug 21, 2024, 5:46 PM

#

exciting

flint idol Aug 21, 2024, 5:47 PM

#

the IPI was sent in the function that's supposed to halt all cpus

#

which is called on panic

#

oh wait nvm

#

it was while the idle task was idling on cpu 3

#

I'm sure it's fine...

#

and I got another one...

#

@real pecan Am I getting the GSI for a PCI pin from uACPI properly here:

uacpi_namespace_node* pciBus = nullptr;
    uacpi_find_devices("PNP0A03", pci_bus_match, &pciBus);
    uacpi_pci_routing_table *pci_routing_table = nullptr;
    uacpi_get_pci_routing_table(pciBus, &pci_routing_table);
    for (size_t i = 0; i < pci_routing_table->num_entries; i++)
    {
        if (pci_routing_table->entries[i].pin != HbaIrqNumber)
            continue;
        if (pci_routing_table->entries[i].source == 0)
            HbaIrqNumber = pci_routing_table->entries[i].index;
        else
        {
            uacpi_resources* resources = nullptr;
            uacpi_get_current_resources(pci_routing_table->entries[i].source, &resources);
            HbaIrqNumber = 
                (resources->entries[pci_routing_table->entries[i].index].type == UACPI_RESOURCE_TYPE_IRQ) ?
                resources->entries[pci_routing_table->entries[i].index].irq.irqs[0] :
                resources->entries[pci_routing_table->entries[i].index].extended_irq.irqs[0];
            uacpi_free_resources(resources);
        }
        break;
    }
    uacpi_free_pci_routing_table(pci_routing_table);```

#

HbaIrqNumber is a pin number based from zero

#

0=A
1=B
2=C
3=D

#

to the managarm source I go nooo

real pecan Aug 21, 2024, 6:03 PM

#

Yeah check managarm or ask @short mortar im on my phone rn

flint idol Aug 21, 2024, 6:03 PM

#

ok

short mortar Aug 21, 2024, 6:04 PM

#

I'm also on phone lol

flint idol Aug 21, 2024, 6:05 PM

#

from that I get GSI 20

#

but the only IRQs on the IOAPIC I see are 10 and 16

#

managarm does the same thing

#

except it indexes zero in the routing table

#

*resource table

#

to the ACPI spec I go

#

which I assume is the index field of the pci route table

short mortar Aug 21, 2024, 6:10 PM

#

There is a good freebsd page that explains it

#

Idk if you might have seen it already

flint idol Aug 21, 2024, 6:11 PM

#

I'll search for it

#

https://people.freebsd.org/~jhb/papers/bsdcan/2007/article/node5.html

PCI Interrupt Routing (Navigating the Maze)

PCI interrupt Routing (Navigating the Maze)

#

?

short mortar Aug 21, 2024, 6:11 PM

#

Yes (I think)

#

Don't know if it could be of use (it's half-copied from Managram), but I'm doing this https://github.com/ThatMishakov/pmOS/blob/main/devicesd/generic/pci/pci.c#L221 and then this https://github.com/ThatMishakov/pmOS/blob/main/devicesd/generic/pci/pci.c#L588 to resolve the interrupts and it seems to be working on physical hw

vernal chasm Aug 21, 2024, 6:19 PM

#

real pecan Yeah check managarm or ask <@350748255929499649> im on my phone rn

I'm not the only one who reads managarm as manga arm right? O_O

short mortar Aug 21, 2024, 6:19 PM

#

flint idol `HbaIrqNumber` is a pin number based from zero

I think it is in ACPI, but the PCI interrupt pin register (don't remember how it's called) is based from 1

flint idol Aug 21, 2024, 6:20 PM

#

I know

#

when I fetch it, I subtract one to make it the same as what ACPI expects

#

I fixed it

#

I was supposed to use the irq_line field as well as the pin

#

to get the IRQ

#

YEAH

#

I can identify sector count

#

and sector size

#

and get IRQs

#

from the AHCI controller

white mulch Aug 21, 2024, 6:33 PM

#

flint idol I was supposed to use the irq_line field as well as the pin

isn't irq line only for the legacy pic?

flint idol Aug 21, 2024, 6:33 PM

#

idk

#

hold up...

#

I resolved the wrong IRQ

real pecan Aug 21, 2024, 6:35 PM

#

white mulch isn't irq line only for the legacy pic?

Nope

#

It depends on how the table is specified in hardware

#

Its 50/50 from what I've seen

#

Some hw links to a link device, some provides a gsi directly

flint idol Aug 21, 2024, 6:36 PM

#

uacpi_namespace_node* pciBus = nullptr;
    uacpi_find_devices("PNP0A03", pci_bus_match, &pciBus);
    uacpi_pci_routing_table *pci_routing_table = nullptr;
    uacpi_get_pci_routing_table(pciBus, &pci_routing_table);
    for (size_t i = 0, pi = 0; i < pci_routing_table->num_entries; i++, pi++)
    {
        if (pci_routing_table->entries[i].pin != HbaIrqNumber)
            continue;
        if (pi != PCINode.irq.int_line)
            continue;
        if (pci_routing_table->entries[i].source == 0)
            HbaIrqNumber = pci_routing_table->entries[i].index;
        else
        {
            uacpi_resources* resources = nullptr;
            uacpi_get_current_resources(pci_routing_table->entries[i].source, &resources);
            HbaIrqNumber = 
                (resources->entries[0].type == UACPI_RESOURCE_TYPE_IRQ) ?
                resources->entries[0].irq.irqs[0] :
                resources->entries[0].extended_irq.irqs[0];
            uacpi_free_resources(resources);
        }
        break;
    }
    uacpi_free_pci_routing_table(pci_routing_table);```

#

I need to look at this code

#

until I see what's causing me to get the wrong GSI

#

I know the GSI should be 16

#

but instead it's 20

#

and the time it "worked" it was zero

#

which was the hpet IRQ

#

I think I know

#

why are there so many entries

#

and how do I know which to choose

#

Each object contains four members that describe the routing for a single PCI interrupt. The first member is an ACPI PCI address using the same format as _ADR. Thus, the upper four bytes contain the slot, and the lower four bytes contain the function. Since the PCI function is not part of a PCI interrupt's address, it is always specified as a wildcard value of 0xFFFF and should be ignored by the operating system. The second member is a single byte indicating the intpin. A value of 0 specifies INTA#, 1 specifies INTB#, etc.

#

from some free bsd thing

white mulch Aug 21, 2024, 6:53 PM

#

you should remove all the ifs (except the last one that checks for the source + the else for that) and add an if to check the routing's pci device + routing's irq pin to make sure that they match what you are trying to find and if they match then continue with the other ifs (and if there is a source then you want to actually loop through the resources instead of blindly assuming that the first one is even an irq)

flint idol Aug 21, 2024, 6:54 PM

#

I'm pretty sure I don't loop, I just use the index in _PRT

#

for the last part

flint idol Aug 21, 2024, 6:55 PM

#

flint idol

based off this

white mulch Aug 21, 2024, 6:55 PM

#

oh yeah, ig I have been doing it wrong KEKW

flint idol Aug 21, 2024, 6:57 PM

#

my irq interface is selling

#

(translation: my irq interface broke)

#

it's causing double + triple faults

#

stack overflow it seems

#

SP=0000:fffffeffffffffd8

flint idol Aug 21, 2024, 7:21 PM

#

fixed that

flint idol Aug 21, 2024, 7:42 PM

#

AHCI is terrible

#

*The AHCI driver

#

it occasionally hangs

#

during init

#

how fun right

flint idol Aug 21, 2024, 7:44 PM

#

flint idol during init

because no irq ever gets sent

#

and as soon as I turn on ahci tracing it goes away

#

ok there it is

flint idol Aug 21, 2024, 8:03 PM

#

I occasionally get a (false positive?) KASAN violation

#

while accessing a command header

haughty abyss Aug 21, 2024, 8:04 PM

#

false positive KASAN violation

#

seems legit man

flint idol Aug 21, 2024, 8:04 PM

#

it's my implementation of kasan

#

so it sucks

#

it was probably non-zeroed memory from the allocator

#

*physical memory

#

that was reallocated after it was freed

flint idol Aug 21, 2024, 8:11 PM

#

flint idol because no irq ever gets sent

it's not because of that

#

it's because sometimes an IRQ gets sent, but port->ci is still set

#

so something in the init waiting on a command to finish never gets signalled

#

because there aren't any IRQs after

#

I think the bug was because I never initialized the event

#

for the command

#

that's set when the command is done

chilly osprey Aug 21, 2024, 8:39 PM

#

vernal chasm I'm not the only one who reads managarm as manga arm right? O_O

managram

#

.!t managarm

umbral iceBOT Aug 21, 2024, 8:39 PM

#

It's Managarm, not Managram

flint idol Aug 21, 2024, 8:45 PM

#

#

it works

#

I need to implement reading and writing

#

and that's really all I have to implement for the AHCI driver (in the sense of what the kernel needs)

#

The kernel has decided it wants to triple fault during smp init on real hw

vernal chasm Aug 21, 2024, 8:59 PM

#

chilly osprey managram

wut

flint idol Aug 21, 2024, 9:32 PM

#

It seems to be because of the new pmm changes

#

confirmed after setting qemu's memory to 4G + 1 mib

#

but it only ever triple faults in smp init...

#

could be because cr3 is allocated in a 64-bit memory region

#

changing it to use the 32-bit version of the pmm allocate function causes it to not triple fault

flint idol Aug 21, 2024, 10:26 PM

#

There is a bug with the ahci driver on real hw

#

I don't get an irq

#

It's probably more like I get an irq but resolved the wrong gsi

#

lemme see what lspci throws at me

#

it says IRQ 29

#

but idk if that's a gsi or just some random number chosen by linux

#

I will now become an ASL interpreter

#

to maybe find where I messed up

#

into the dsdt I go!

#

I think I got 0x11

#

Package (0x04)
{
    0x0001FFFF, 
    One, 
    Zero, 
    0x11
}, ```

#

address.slot=0x1f
address.function=ANY
pin=PINB#
source=0
index=0x11 (GSI to connect the thingy to)

#

I match this:

Package (0x04)
{
    0x001FFFFF, 
    Zero, 
    LNKF, 
    Zero
}, ```

#

LNKF is confusing

short mortar Aug 21, 2024, 10:51 PM

#

flint idol It's probably more like I get an irq but resolved the wrong gsi

I think you can look at PCI status register to see if the device raises it

flint idol Aug 21, 2024, 10:51 PM

#

you see I would, except every cpu is on the idle task

#

so I can't

#

without some funny buisness

short mortar Aug 21, 2024, 10:52 PM

#

Can you not busy wait?

flint idol Aug 21, 2024, 10:52 PM

#

I can

#

but that's boring

flint idol Aug 21, 2024, 10:52 PM

#

flint idol I can

maybe not

#

I currently use WaitOnObject to wait on an event that's set on command completion

#

which blocks the current thread

#

I'll add logs to the ahci irq handler

flint idol Aug 21, 2024, 10:54 PM

#

short mortar Can you not busy wait?

you said you had some real hw where the AHCI GSI was 19, did it use an intel cpu or amd cpu

short mortar Aug 21, 2024, 10:54 PM

#

flint idol It's probably more like I get an irq but resolved the wrong gsi

I had this problem and it was because of QEMU ignored polarity and active low stuff and it worked there

short mortar Aug 21, 2024, 10:54 PM

#

flint idol you said you had some real hw where the AHCI GSI was 19, did it use an intel cpu...

I got 19 on intel and 33 on Ryzen (I think)

flint idol Aug 21, 2024, 10:55 PM

#

hmmm

#

I got 19 on my intel computer

#

probably something to do with the chipset

#

let me guess, the ahci controller was on slot 31, function two on the intel machine

short mortar Aug 21, 2024, 10:55 PM

#

Idk

#

I can check...

flint idol Aug 21, 2024, 10:56 PM

#

nah it doesn't matter

short mortar Aug 21, 2024, 10:59 PM

#

Do you have other IOAPIC interrupts?

#

Working

flint idol Aug 21, 2024, 11:03 PM

#

indeed

real pecan Aug 21, 2024, 11:36 PM

#

flint idol `LNKF` is confusing

Look at linkf resources

flint idol Aug 21, 2024, 11:37 PM

#

I did

real pecan Aug 21, 2024, 11:37 PM

#

What do they say

flint idol Aug 21, 2024, 11:38 PM

#

Let me check

#

On my phone rn

#

Method (_CRS, 0, Serialized)  // _CRS: Current Resource Settings
{
    Name (RTLF, ResourceTemplate ()
    {
        IRQ (Level, ActiveLow, Shared, )
            {}
    })
    CreateWordField (RTLF, One, IRQ0)
    IRQ0 = Zero
    IRQ0 = (One << (PFRC & 0x0F))
    Return (RTLF) /* \_SB_.LNKF._CRS.RTLF */
}

real pecan Aug 21, 2024, 11:39 PM

#

Ah ok

flint idol Aug 21, 2024, 11:40 PM

#

the numbers, what do they mean

real pecan Aug 21, 2024, 11:40 PM

#

I guess it just reads the gsi from some hw reg

flint idol Aug 21, 2024, 11:41 PM

#

SRS seems to write to said register

real pecan Aug 21, 2024, 11:41 PM

#

Makes sense

flint idol Aug 21, 2024, 11:41 PM

#

Method (_SRS, 1, Serialized)  // _SRS: Set Resource Settings
{
    CreateWordField (Arg0, One, IRQ0)
    FindSetRightBit (IRQ0, Local0)
    Local0--
    PFRC = Local0
}```

#

_STA reads from it to get the link status

real pecan Aug 21, 2024, 11:42 PM

#

I see

flint idol Aug 21, 2024, 11:42 PM

#

flint idol I currently use `WaitOnObject` to wait on an event that's set on command complet...

the command never completes

#

after temporarily switching to a busy loop

#

I've heard vbox's ahci emulation is better than qemu's

#

and behold; it hangs

#

last vbox log before it hangs is:

AHCI#0: Reset the HBA```

flint idol Aug 22, 2024, 1:59 AM

#

from what I've seen so far

#

I seem to have resolved the wrong IRQ number / have a bug with the IOAPIC implementation

#

I get HPET irqs

#

so it's not the IOAPIC

flint idol Aug 22, 2024, 3:27 AM

#

after some more debugging

#

it's a problem with my hba init

inland radish Aug 22, 2024, 5:03 AM

#

@flint idol does the AHCI controller not support MSI / MSI-X

short mortar Aug 22, 2024, 10:38 AM

#

inland radish <@841762704766140486> does the AHCI controller not support MSI / MSI-X

I would imagine it probably does

inland radish Aug 22, 2024, 10:40 AM

#

So then he should consider using that

flint idol Aug 22, 2024, 11:49 AM

#

inland radish <@841762704766140486> does the AHCI controller not support MSI / MSI-X

Pretty sure it does

short mortar Aug 22, 2024, 12:33 PM

#

But where's fun in that

flint idol Aug 22, 2024, 12:33 PM

#

exactly

#

(I don't want to implement MSI(-X))

short mortar Aug 22, 2024, 12:34 PM

#

Also MSI doesn't work everywhere

flint idol Aug 22, 2024, 12:39 PM

#

uhhh yeah that too

inland radish Aug 22, 2024, 12:45 PM

#

flint idol (I don't want to implement MSI(-X))

it's easy man

flint idol Aug 22, 2024, 12:45 PM

#

Fine. I'll take a look after I fix this bug with AHCI

inland radish Aug 22, 2024, 12:45 PM

#

msi is basically

flint idol Aug 22, 2024, 12:45 PM

#

Kernel Panic on CPU 1 in thread 6, owned by process 1. Reason: OBOS_PANIC_DRIVER_FAILURE. Information on the crash is below:
Port physical layer not online after reset.

inland radish Aug 22, 2024, 12:46 PM

#

device writes 32-bit payload to address

vale nymph Aug 22, 2024, 12:46 PM

#

I only support msi(x), I very much prefer that since it doesnt depend on having acpi working

#

The device doesnt support it? Your computer is bad, sorry

inland radish Aug 22, 2024, 12:47 PM

#

i also prefer it because it allows a very nice integration with interrupt layering

inland radish Aug 22, 2024, 12:47 PM

#

inland radish device writes 32-bit payload to address

that's the gist of it lol

flint idol Aug 22, 2024, 12:47 PM

#

oh well that's nice I guess

inland radish Aug 22, 2024, 12:47 PM

#

that address is usually an address in LAPIC MMIO

flint idol Aug 22, 2024, 12:47 PM

#

is it x86 specific though

vale nymph Aug 22, 2024, 12:47 PM

#

The data and address are arch specific

#

The concept isnt iirc

inland radish Aug 22, 2024, 12:48 PM

#

i think its PCI specific

short mortar Aug 22, 2024, 12:48 PM

#

vale nymph The device doesnt support it? Your computer is bad, sorry

It doesn't work on RISC-V in QEMU

inland radish Aug 22, 2024, 12:48 PM

#

and not necessarily specific to the arch

flint idol Aug 22, 2024, 12:48 PM

#

ok

#

I know how I'm going to abstract interrupts to drivers further. I would've thought an irq interface which dispatches your IRQs would be good enough, guess not.

inland radish Aug 22, 2024, 12:49 PM

#

I'm pretty sure it's good enough

flint idol Aug 22, 2024, 12:51 PM

#

basically I should prefer MSI over the IOAPIC, and MSI-X over MSI

#

and if none of those exist, yell at the driver

#

(return OBOS_STATUS_INVALID_OPERATION)

inland radish Aug 22, 2024, 12:52 PM

#

i think ahci controllers that operate only via legacy IRQs are golden eggs

flint idol Aug 22, 2024, 12:52 PM

#

damn this is actually quite simple

#

set message address to LAPIC

vale nymph Aug 22, 2024, 12:53 PM

#

flint idol and if none of those exist, yell at the driver

Leave a thread polling constantly meme

flint idol Aug 22, 2024, 12:54 PM

#

although idk what these are supposed to be

#

nvm

#

I think I set mask to one to mask the IRQ

#

and pending is just to say whether there is an irq pending or not

#

If capable, you can mask individual messages by setting the corresponding bit (1 << n), in the mask register.

If a message is pending, then the corresponding bit in the pending register is set.

#

From the osdev wiki

flint idol Aug 22, 2024, 3:55 PM

#

I think I got MSI implemented

#

idk if it works yet

#

time to go back to debugging AHCI nooo

#

and this is why you don't write AHCI code at 12 in the morning...

#

so MSI works...

#

... except I'm checking if there was an IRQ wrong

#

I'm checking if bit 3 of the PCI status register is set

#

but it isn't for some reason

#

after the IRQ

#

Interrupt Status - Represents the state of the device's INTx# signal. If set to 1 and bit 10 of the Command register (Interrupt Disable bit) is set to 0 the signal will be asserted; otherwise, the signal will be ignored.

#

which means I need another way

#

I can check the pending bit for MSI(-X)

#

but as for the IOAPIC fallback

#

I'll have a weak function that the arch defines if it can check the IRQ status

#

otherwise it's up to the driver to check the IRQ status

#

if the IRQ checker callback is nullptr

#

guess I have to always use the devices IRQ checker

#

so I can receive MSI IRQs

#

I have no luck receiving an IRQ on vbox

#

maybe if I check what @ infy's OS does

#

because apparently that code works

#

PxIE does permit all irqs

#

same thing with GhcIe

#

maybe the PCI command register's IRQ disable bit is set

#

doesn't seem like it

#

it isn't set

#

at all

#

or maybe I don't know how to read

#

#

I was accessing command at status' offset

#

and vice versa

#

in the AHCI driver

flint idol Aug 22, 2024, 5:09 PM

#

I have confirmed it's not my IRQ interface

#

it is my AHCI driver

short mortar Aug 22, 2024, 5:30 PM

#

flint idol > Interrupt Status - Represents the state of the device's INTx# signal. If set t...

I think the spec said that MSI bypassed that stuff

flint idol Aug 22, 2024, 5:31 PM

#

idk

#

but I now check it using the HBA register

short mortar Aug 22, 2024, 5:31 PM

#

flint idol

So this bit is what pulls the interrupt line down

#

So like I think you should be checking IS register?

flint idol Aug 22, 2024, 5:32 PM

#

I do

short mortar Aug 22, 2024, 5:32 PM

#

(I haven't tried vbox, but my code seems to be working on QEMU and 3 PCs)

flint idol Aug 22, 2024, 5:32 PM

#

what about IRQs

#

I've heard that vbox has better IRQ emulation than qemu

short mortar Aug 22, 2024, 5:33 PM

#

With IRQ routing

flint idol Aug 22, 2024, 5:33 PM

#

so I'm using it instead of testing on real hw each time

short mortar Aug 22, 2024, 5:33 PM

#

flint idol I've heard that vbox has better IRQ emulation than qemu

VBox doesn't have ECAM

flint idol Aug 22, 2024, 5:33 PM

#

I meant

#

AHCI emulation

flint idol Aug 22, 2024, 5:34 PM

#

short mortar VBox doesn't have ECAM

not a problem when you can just assume the PCI bus existence meme

short mortar Aug 22, 2024, 5:34 PM

#

short mortar VBox doesn't have ECAM

And I haven't implemented legacy PCI access to try it

#

Have you tried VMware?

flint idol Aug 22, 2024, 5:43 PM

#

No

flint idol Aug 22, 2024, 6:20 PM

#

OBOS on real hw hangs during HBA init

#

specifically during port init

#

I think I might git revert my HBA init

#

because I rewrote it at midnight when I was very tired

#

guess I'm rewriting HBA initialization code now

#

Enable interrupts, DMA, and memory space access in the PCI command register
Memory map BAR 5 register as uncacheable.
Perform BIOS/OS handoff (if the bit in the extended capabilities is set)
Reset controller
Register IRQ handler, using interrupt line given in the PCI register. This interrupt line may be shared with other devices, so the usual implications of this apply.
Enable AHCI mode and interrupts in global host control register.
Read capabilities registers. Check 64-bit DMA is supported if you need it.
For all the implemented ports:
Allocate physical memory for its command list, the received FIS, and its command tables. Make sure the command tables are 128 byte aligned.
Memory map these as uncacheable.
Set command list and received FIS address registers (and upper registers, if supported).
Setup command list entries to point to the corresponding command table.
Reset the port.
Start command list processing with the port's command register.
Enable interrupts for the port. The D2H bit will signal completed commands.
Read signature/status of the port to see if it connected to a drive.
Send IDENTIFY ATA command to connected drives. Get their sector size and count.

#

seems simple enough

#

I think I'll follow what the spec says instead...

short mortar Aug 22, 2024, 6:35 PM

#

flint idol > Enable interrupts, DMA, and memory space access in the PCI command register > ...

The infy's post also follows that I think

short mortar Aug 22, 2024, 6:37 PM

#

flint idol > Enable interrupts, DMA, and memory space access in the PCI command register > ...

The wiki seems incomplete..?

flint idol Aug 22, 2024, 6:37 PM

#

probably is, which is why I'm reading the spec instead

short mortar Aug 22, 2024, 6:38 PM

#

short mortar The infy's post also follows that I think

https://forum.osdev.org/viewtopic.php?t=40969

flint idol Aug 22, 2024, 6:40 PM

#

@infy has saved me twice, once with uACPI, and now twice with the AHCI init

#

~~when uAHCI~~

vale nymph Aug 22, 2024, 6:41 PM

#

uKernel

real pecan Aug 22, 2024, 6:43 PM

#

short mortar https://forum.osdev.org/viewtopic.php?t=40969

Oh yeah I remember this torture

#

Lmao

#

Glad to be of service

short mortar Aug 22, 2024, 6:44 PM

#

I think there needs to be some sort of agreement for universal driver format for hobby OSes

flint idol Aug 22, 2024, 6:45 PM

#

mine

#

why?

#OBOS (not vibecoded)

eventually

eventually

eventually