OBOS (not vibecoded) | Operating System Development | Page 17

flint idol Aug 27, 2024, 11:53 AM

#

silliness

#

mischief

#

nonsense

#

foolishness

#

I got the FAT driver to traverse directories

flint idol Aug 27, 2024, 12:28 PM

#

I can probably mount FAT partitions now

#

as I've implemented all the functions in the fat driver needed

flint idol Aug 27, 2024, 12:53 PM

#

#

I can mount fat partitions

#

!!!!

#

I wonder how this would fare on real hw

#

inb4 it corrupts my disk

#

by some miracle

#

or it just hangs, that might happen too

#

It crashes

#

But that's better than hanging

#

cache->root_sector = cache->fatType == FAT32_VOLUME ? ClusterToSector(cache, bpb->ebpb.fat32.rootCluster) : (FirstDataSector-RootDirSectors);

#

it crashes there

#

on a nullptr access

#

according to ubsan, it was here: ClusterToSector(cache, bpb->ebpb.fat32.rootCluster)

#

if bpb were nullptr, it would've crashed eariler

#

same thing with cache

#

#define ClusterToSector(cache, n) (((n - 2) * (cache)->bpb->sectorsPerCluster) + (cache)->FirstDataSector)

#

cache->bpb was nullptr

#

I didn't notice it before since my disk image used FAT16

#

and fat16's root directory sector is at a fixed offset which didn't need to use cache->bpb

flint idol Aug 27, 2024, 1:01 PM

#

flint idol `cache->bpb` was nullptr

the solution is to simply initialize this before that line

#

Now I get an asan violation because of a buffer overflow

#

It page faults now

#

Annoyingly

real pecan Aug 27, 2024, 1:09 PM

#

lmao

#

any new obos feature comes with 10 hours of debugging hangs and faults

#

but thats a price for stability of obos

flint idol Aug 27, 2024, 1:09 PM

#

Because now I will need to disable kasan to get an accurate rip

#

Indeed

#

The paths are also messed up

#

I think what I'll do is copy the partition from my disk into a file

#

Then debug it in qemu

#

The paths are also messed up

#

if all goes well, I can load the disk image as a raw partition in my kernel

#

if all does not go well I need to make the disk image contain a partition table

#

INVALID_ARGUMENT

#

probably from the mbr parser

#

if I remove the mbr signature it won't happen

#

that doesn't work

#

time to handwrite an mbr

#

I think I did that correctly

#

aaand the kernel doesn't like it

#

I accidentally put in a number as big-endian

flint idol Aug 27, 2024, 1:49 PM

#

I have been able to reproduce the crash

#

fuck

#

the paths are now normal

#

it was a bug with my 8.3 filename parsing

#

seems to be some funny buisness with the LFN entries

#

that's causing the pf

flint idol Aug 27, 2024, 2:04 PM

#

flint idol seems to be some funny buisness with the LFN entries

(along with an allocator bug in Reallocate)

#

fixed

#

other than some bugs related to paths

#

it works

flint idol Aug 27, 2024, 2:27 PM

#

this doesn't make sense

#

the LFN entries are fucked

#

on this partition

#

why are there just random characters

flint idol Aug 27, 2024, 2:56 PM

#

I've been able to remove the special characters by ensuring that any char < 0 is dropped when the LFN entry chain is being processed

#

but there are still some extraneous characters

#

which do not appear anywhere else

#

and also there are seemingly two directory entries which simply do not exist

#

but other than some schizophrenic moments in the fat driver

#

as I stated above

#

I just need to implement reading

#

writing

#

and some other stuff

#

(making files, removing files, and moving files iirc)

#

and also setting file perms

flint idol Aug 27, 2024, 7:50 PM

#

the fat driver mysteriously hangs

#

on real hw

#

while probing the partition

#

which I have not been able to reproduce on qemu

#

what I should do is just give qemu my entire disk

#

what could possibly go wrong?

#

nothing

#

*nothing has went wrong

#

as long as I haven't tested writing anywhere and forgot, I'll be fine

#

is there such thing as AHCI passthrough

#

actually maybe that's not a good idea...

#

I'll just switch to a tty and kill the other user session along with the X server

#

and basically anything

#

and pray I don't lock up my system

#

and corrupt shit because a write was going on

#

what could go wrong if I simply print each time an object is waited on

#

probably nothing right

#

Oh nvm it suddenly doesn't hang anymore

#

But it does hang without the log

#

An dit hangs in seemingly the same place each time

#

*and it

#

Nvm it doesn't

#

Likely a synchronization issue with IRQs and the event object [in the ahci driver]

#

Well more like dpc handler

#

I think I should go to IRQL_DISPATCH before blocking the thread

#

then lower after the thread is unblocked

#

so that no DPCs run and wake the event right after it is unlocked

#

That fixed it

vernal chasm Aug 28, 2024, 9:11 AM

#

Did nothing else yesterday? Odd

flint idol Aug 28, 2024, 10:09 AM

#

vernal chasm Did nothing else yesterday? Odd

My yesterday was filled with bug fixes

vernal chasm Aug 28, 2024, 10:17 AM

#

flint idol My yesterday was filled with bug fixes

Oof

flint idol Aug 28, 2024, 2:25 PM

#

I think I know a solution to my LFS problem

#

before, I was making sure there is only one LFS entry that is the "last" one

#

and I used the order to get the size

#

instead of that, I'll just allow truncation of the list

#

so that no garbage LFN entries show up

#

causing weird filenames

#

that fixed that bug

#

but now I am welcomed by:

Page fault at 0xffffffff80005dea in kernel-mode while to write page at 0xffffff0000444000, which is unpresent. Error code: 2```

#

fixed

#

other than the fat driver hallucinating some stuff

#

in the filenames

#

it's nearly done

#

just need reading/writing

#

tbh I'm surprised mounting the partition went so smoothly

#

it was completely untested

#

I'm pretty sure it was the same thing with mounting in a directory

#

time to implement reading

#

which will be easy

#

for each cluster: read into buffer at current position

#

writing won't be as easy

#

but it won't be hard

#

basically:
LOCK FAT
if file is truncated by write: free extra clusters and change filesize
if file is expanded: free old clusters, allocate 'n' NEW clusters, change filesize
flush FAT and DIRENT
UNLOCK FAT
write data at appropriate cluster offset and byte offset
flush the dirty clusters
return SUCCESS

#

creating a file:
expand the parent directory if needed
make LFN entries
make 8.3 filename
setup dirent to "default" settings
add cache entry
return SUCCESS

#

moving a file:
create new file in the new parent directory
delete the old dirent
move old dirent contents (e.g., filesize, cluster begin, etc.) to the new dirent
move cache entry to new parent directory, and change the filename and path
return SUCCESS

#

setting file perms:
set file perms in cache entry
if the write bit is disabled anywhere in the perms, set READ_ONLY in the file attributes
flush any changes to disk

#

removing files:
remove cache entry
free clusters
nuke dirents, free clusters taken by dirent

flint idol Aug 28, 2024, 3:24 PM

#

reading should be implemented

#

it shouldn't

#

I need to seek

#

now it should work

#

almost forgot something

#

I find it funny that I wrote basically the entirety of the scheduler in ~3 commits

#

with most in one commit

#

in total it was modified in 24 commits according to git blame

#

aaaaaand reading doesn't work

#

I can read from FAT volumes

real pecan Aug 28, 2024, 4:03 PM

#

now test fat12 and 16

flint idol Aug 28, 2024, 4:03 PM

#

it is a FAT16 volume

real pecan Aug 28, 2024, 4:03 PM

#

now test fat12 and 32

flint idol Aug 28, 2024, 4:05 PM

#

since there is no difference between fat12/fat32 in terms of reading clusters, I think I should be fine

#

but I'll test FAT32/12 anyway

real pecan Aug 28, 2024, 4:05 PM

#

well root dir is different for 12

flint idol Aug 28, 2024, 4:06 PM

#

no, it's the same as FAT12

#

iirc

#

it's different on FAT32

real pecan Aug 28, 2024, 4:06 PM

#

no

#

only fat12 has a fixed root directory

flint idol Aug 28, 2024, 4:07 PM

#

I'm checking the spec rn

real pecan Aug 28, 2024, 4:07 PM

#

nvm

#

im rarted

#

yeah its only different for 32

flint idol Aug 28, 2024, 4:07 PM

#

the only place where I should have a difference between the three fat types is in writing

#

because the FAT format is different

flint idol Aug 28, 2024, 4:38 PM

#

Time to implement the cluster allocator thing

#

sounds pretty simple

#

uint32_t AllocateClusters(fat_cache* volume, size_t nClusters);
// Returns true if the cluster region was extended, otherwise you need to reallocate the clusters.
bool ExtendClusters(fat_cache* volume, uint32_t cluster, size_t nClusters);
void TruncateClusters(fat_cache* volume, uint32_t cluster, size_t newClusterCount, size_t oldClusterCount);
void FreeClusters(fat_cache* volume, uint32_t cluster, size_t nClusters);```

#

I've written some stuff for the cluster allocator

#

time to implement it

#

I wonder if it'd be worth it to make a freelist of some sorts at volume-probe

#

to make cluster allocation easier

flint idol Aug 28, 2024, 5:35 PM

#

flint idol I wonder if it'd be worth it to make a freelist of some sorts at volume-probe

probably not

#

anyway I'm going to cache the entire fat into memory

flint idol Aug 28, 2024, 6:22 PM

#

I have implement cluster allocation

#

and freeing

flint idol Aug 28, 2024, 6:23 PM

#

flint idol ```c uint32_t AllocateClusters(fat_cache* volume, size_t nClusters); // Returns ...

basically these

#

I will not be testing obos on real hw

#

until I am 10000% sure it's stable

#

and won't crash or hang in the middle of a write

#

or won't write to some random sector

flint idol Aug 28, 2024, 6:52 PM

#

I'm testing writing

#

rn

#

if all goes well, I should be able to write to a file

#

and see the changes on the disk image using mcopy

#

Assertion failed in function VfsH_PCDirtyRegionCreate. File: /home/oberrow/Code/obos/src/oboskrnl/vfs/pagecache.c, line 54. !(off >= pc->sz || (off+sz) >= pc->sz)```

#

t'was trying to make the dirty region before resizing the page cache appropriately

#

Assertion failed in function Vfs_FdWrite. File: /home/oberrow/Code/obos/src/oboskrnl/vfs/fd.c, line 152. obos_expect(dirty != nullptr, 0)```

#

fixed

#

but something funky is happenning

#

the thing was using the page cache

#

for writes

#

then on close it was flushing the page cache

#

actually nvm

#

it was probably just the thing working properly

#

and the write is not on the disk image

#

rather the write failed, or qemu doesn't want to make the write be on the disk image

#

or I could've just corrupted some random block, you never know

#

or the loop device could be out of date

#

it worked!

#

the loop device was stale it seems

#

and larger writes break

#

why? no idea.

#

oops

#

shit

#

I corrupted the fat

#

so good at this

real pecan Aug 28, 2024, 7:21 PM

#

fsck.fat or whatever

flint idol Aug 28, 2024, 7:21 PM

#

nah it was on the qemu disk image

#

for which I made a backup

#

so I didn't have to remake it

#

even if I didn't

#

there is always git revert

real pecan Aug 28, 2024, 7:22 PM

#

Fsck prints out how you corrupted it, its not necessarily for fixing anything

flint idol Aug 28, 2024, 7:22 PM

#

in that case

vale nymph Aug 28, 2024, 7:23 PM

#

The fsck tools are your best friends when writing filesystem drivers

#

e2fsck helped me find vnode refcounting issues of all things

flint idol Aug 28, 2024, 7:24 PM

#

vale nymph The fsck tools are your best friends when writing filesystem drivers

they're also your friend when you corrupt 4G at the beginning of your hard drive with a backup of like everything you have

#

#filesystems message

#

wait if a bit is set in the fat

#

it's allocated, right?

#

I think so

real pecan Aug 28, 2024, 7:37 PM

#

Thats not how it works

flint idol Aug 28, 2024, 7:37 PM

#

it's not

#

?

real pecan Aug 28, 2024, 7:37 PM

#

U mask out the 28 bits then check them

#

Or whatever the bitness of the fat

flint idol Aug 28, 2024, 7:38 PM

#

real pecan U mask out the 28 bits then check them

so if a bit is one

#

in the fat entry

#

the cluster is allocated?

real pecan Aug 28, 2024, 7:38 PM

#

Depends which bit

flint idol Aug 28, 2024, 7:39 PM

#

flint idol wait if a bit is set in the fat

*in a fat entry

#

do I have to write to all FATs when I modify it

flint idol Aug 28, 2024, 8:04 PM

#

I almost™️ can write to files

#

#

I absolutely deep fried the image

flint idol Aug 28, 2024, 8:22 PM

#

flint idol I absolutely deep fried the image

my cached FAT is all zero...

#

oops

#

and I deep fried the allocator structs

#

other than the fact that I deep fry the FAT on each write

#

I can write files with my FAT driver

#

cache->fat = OBOS_NonPagedPoolAllocator->ZeroAllocate(OBOS_NonPagedPoolAllocator, fatSz*blkSize, sizeof(uint8_t), nullptr);
uint64_t fat_base = GetFatEntryAddrForCluster(cache, 0).lba;
Vfs_FdSeek(cache->volume, fat_base*blkSize, SEEK_SET);
for (uint64_t sec = fat_base, offset = 0; sec < fat_base+fatSz; sec++, offset += blkSize)
    Vfs_FdRead(cache->volume, cache->fat + offset, blkSize, nullptr);

#

wonder what's wrong with my code to load the fat

#

into memory

#

fat_entry_addr GetFatEntryAddrForCluster(fat_cache* cache, uint32_t cluster)
{
    uint32_t fatOffset = 0;
    switch (cache->fatType) {
        case FAT32_VOLUME:
            fatOffset = cluster*2;
            break;
        case FAT16_VOLUME:
            fatOffset = cluster*4;
            break;
        case FAT12_VOLUME:
            fatOffset = cluster + (cluster / 2);
            break;
        default:
            OBOS_ASSERT(false && "Invalid fat type.");
            break;
    }
    return (fat_entry_addr){
        .lba = (cache->bpb->reservedSectorCount + fatOffset / cache->bpb->bytesPerSector),
        .offset = fatOffset % cache->bpb->bytesPerSector
    };
}```

#

maybe I need to learn how to read

#

I was calculating fatOffset wrong

#

so I was reading from the wrong sector

flint idol Aug 28, 2024, 9:09 PM

#

I fixed that bug it seems

#

but I still can't write properly

#

because I can't extend the cluster chain properly

#

the heap gets deep fried for some reason

#

corrupted beyond anything

#

and the corrupted region is filled with the filler text being used to test the fat driver

#

and the corruption happens after a memcpy

#

I'm going to make memcpy call __asan_load_n

#

(it's in asm, so it doesn't do that automatically)

#

I fixed it

#

although the fat is still scuffed

#

mainly because I had no idea how the FAT works

flint idol Aug 28, 2024, 11:30 PM

#

for some reason the fat is not being read properly into memory

#

(again)

#

it reads the first sector of the fat

#

but the rest is all zero

#

despite it not being like that

#

I have a couple words:
FUCK AHCI

flint idol Aug 29, 2024, 12:00 AM

#

WAIT I MIGHT BE STUPID

#

I WAS USING THE LBA

#

TO OFFSET THE FAT

#

CACHE

#

DRTGFYHUTGFJUHRFTYH

flint idol Aug 29, 2024, 12:18 AM

#

how

#

the

#

hell

#

does

#

fat allocation

#

work

#

nooo

#

after RTFM

#

The list of free clusters in the FAT is nothing more than the list of all clusters that contain the value 0
in their FAT cluster entry.

#

might rename my fat driver to slowfat

flint idol Aug 29, 2024, 1:34 AM

#

I think writing works now

#

well, almost works

#

other than:

/file.txt
  File size is 4477 bytes, cluster chain length is > 5120 bytes.
  Truncating file to 4477 bytes.
Reclaimed 3405 unused clusters (3486720 bytes).```

#

it works

flint idol Aug 29, 2024, 1:54 AM

#

something

#

somewhere

#

keeps on corrupting everything

#

I fixed that

#

idk where I'd be if it weren't for KASAN

ashen goblet Aug 29, 2024, 2:06 AM

#

what kind of issues does it help you with mostly? over/underflows?

#

use after frees?

flint idol Aug 29, 2024, 2:06 AM

#

use after frees and overflows

#

I haven't implemented anything to help with underflows

#

I fixed the heap corruption

#

but two writes to the same file causes the fat to be corrupted or something like that:

oberrow@Acer-AIO:~/Code/obos/scripts$ sudo fsck.fat /dev/loop0
fsck.fat 4.2 (2021-01-31)
/file.txt
  Contains a free cluster (39399). Assuming EOF.
/file.txt
  File size is 4477 bytes, cluster chain length is 0 bytes.
  Truncating file to 0 bytes.
Reclaimed 4650 unused clusters (4761600 bytes).```

#

I was able to fix that

flint idol Aug 29, 2024, 2:18 AM

#

flint idol other than: ``` /file.txt File size is 4477 bytes, cluster chain length is > 5...

to get this once again

#

oh nvm

#

my fat driver knows its ABCs

#

the FAT driver cannot write anything reliably yet

#

more than one 1 write that extends the file will break the driver

#

uint32_t ecx = 0;
__cpuid__(1, 0, nullptr, nullptr, &ecx, nullptr);
bool isHypervisor = ecx & BIT_TYPE(31, UL) /* Hypervisor bit: Always 0 on physical CPUs. */;
if (!isHypervisor)
    OBOS_Panic(OBOS_PANIC_FATAL_ERROR, "no, just no.\n");```

#

added this code in case I try to do something stupid (run the writing test on real hw)

flint idol Aug 29, 2024, 2:40 AM

#

flint idol more than one 1 write that extends the file will break the driver

I fixed this

#

well more like I fixed the writing problem

#

extending the file is still broken

ashen goblet Aug 29, 2024, 2:51 AM

#

flint idol I haven't implemented anything to help with underflows

how are heap overflows dtected? shadow area behind the allocation?

ornate ginkgo Aug 29, 2024, 5:38 AM

#

You can do it that way, or even put an unmapped page after each allocation - that let's you know exactly when the overrun happens

inland radish Aug 29, 2024, 8:06 AM

#

flint idol might rename my fat driver to `slowfat`

you're not the only one who thought of that lmao

inland radish Aug 29, 2024, 8:06 AM

#

flint idol added this code in case I try to do something stupid (run the writing test on re...

consider getting a piece of test hardware you dont store all your important data on

flint idol Aug 29, 2024, 10:16 AM

#

maybe eventually

#

anyway

#

it's time for me to see what kind of delusional code I wrote last night

flint idol Aug 29, 2024, 10:18 AM

#

ashen goblet how are heap overflows dtected? shadow area behind the allocation?

I use a shadow area to detect heap overflows, however idk what a proper asan impl. would do

flint idol Aug 29, 2024, 10:28 AM

#

flint idol but two writes to the same file causes the fat to be corrupted or something like...

this is because the driver is allocating the clusters incorrectly

#

resulting in the FAT entry for that cluster to be all zeroes

#

I am unsure of the cause though

flint idol Aug 29, 2024, 12:31 PM

#

flint idol might rename my fat driver to `slowfat`

because it is slow af

#

I will rename it that

#

and also add:

// abandon all hope ye who enter here```
to each file

flint idol Aug 29, 2024, 12:46 PM

#

flint idol I am unsure of the cause though

it is because something is writing at the wrong offset in the FAT

#

instead of writing at offset 0x1E4, it writes at 0xE4

flint idol Aug 29, 2024, 1:06 PM

#

I MIGHT BE STUPID

#

for (size_t curr = 0; curr < cache->bpb->nFATs; curr++)
    for (uint64_t sec = fat_base; sec < fat_base+cache->fatSz; sec++)
        Vfs_FdWrite(cache->volume, cache->fat, cache->blkSize, nullptr);```

#

see anything wrong with this code

#

possibly?

#

(hint, it's in the second parameter of Vfs_FdWrite)

#

ok I think writing works now

#

time to optimize this:

void FlushFAT(fat_cache* cache)
{
    Core_MutexAcquire(&cache->fd_lock);
    Core_MutexAcquire(&cache->fat_lock);
    // TODO: Mark dirty regions in the functions above to speed this up.
    uint64_t fat_base = cache->bpb->reservedSectorCount;
    Vfs_FdSeek(cache->volume, fat_base*cache->blkSize, SEEK_SET);
    for (size_t curr = 0; curr < cache->bpb->nFATs; curr++)
        for (uint64_t sec = fat_base; sec < fat_base+cache->fatSz; sec++)
            Vfs_FdWrite(cache->volume, cache->fat+((sec-fat_base)*cache->blkSize), cache->blkSize, nullptr);
    Core_MutexRelease(&cache->fat_lock);
}```

#

I'm thinking to just have a dirty FAT entry list

#

and only those sectors of the FAT are flushed

#

because there is no need to flush each sector of the FAT each time

flint idol Aug 29, 2024, 1:36 PM

#

flint idol I'm thinking to just have a dirty FAT entry list

I think I'll use a bitmap instead

flint idol Aug 29, 2024, 2:05 PM

#

flint idol because it is slow af

that isn't even an exageration

#

a write of 1 mib

#

takes ages

#

and then continues to corrupt the file

#

/file.txt
  Contains a free cluster (38739). Assuming EOF.
/file.txt
  File size is 1048576 bytes, cluster chain length is 0 bytes.
  Truncating file to 0 bytes.
Reclaimed 4 unused clusters (4096 bytes).```

#

two things could've happened:

the fat entry truly has nothing at that cluster
the FAT flush code is buggy

#

it's the latter

#

for (uint32_t cluster = 0; !isLastCluster(cache, cluster); cluster++)
{
    if (!(cache->dirty_fat_entries[cluster/8] & BIT(cluster%8)))
        continue;
    // OBOS_Debug("dirty cluster 0x%08x\n", cluster);
    fat_entry_addr addr = {};
    GetFatEntryAddrForCluster(cache, cluster, &addr);
    Vfs_FdSeek(cache->volume, (addr.lba+((i*cache->fatSz)/cache->blkSize))*cache->blkSize, SEEK_SET);
    memcpy(blk, cache->fat + (addr.lba-fat_base)*cache->blkSize, cache->blkSize);
    cluster += (cache->blkSize-(cluster%cache->blkSize)); // skip the rest of the clusters in this sector of the FAT
    Vfs_FdWrite(cache->volume, blk, cache->blkSize, nullptr);
}```

#

I need to look at this really hard

#

and figure out what's wrong

#

currently it flushes only one sector of the FAT

#

*one dirty sector

#

despite there being a lot more

#

hmmm

#

I think I found a problem

flint idol Aug 29, 2024, 8:42 PM

#

FAT driver is getting boring

#

lemme see what was next on my list

#

paging out to files and partitions

flint idol Aug 29, 2024, 8:43 PM

#

flint idol FAT driver is getting boring

it's like done

#

but I need to speed it up

#

but that includes fixing my page cache

#

but then

#

hmmm

#

I think I'll be fixing that

#

and also my AHCI driver

#

FUCK AHCI

#

anyway

#

into the qemu log I go!

#

yeah I got no idea

#

since ddd is a data display debugger

#

I will use its data displaying abilities

#

to examine the physical regions

inland radish Aug 29, 2024, 8:57 PM

#

flint idol since `ddd` is a data display debugger

If you use -g when compiling and are using qemu then you can also use gdb!

flint idol Aug 29, 2024, 8:58 PM

#

I know how to use a debugger thank you

#

ddd is just a gui for gdb

#

what was the syntax to declare a pointer to an array

#

not int**

#

but something like int(*)[2]

#

nvm

barren merlin Aug 29, 2024, 9:01 PM

#

flint idol what was the syntax to declare a pointer to an array

int (*a)[2]

flint idol Aug 29, 2024, 9:01 PM

#

I was using it in gdb

#

turns out you can just do p *array@len

#

well would ya look at that

#

the PRDTs are completely wrong

#

time to debug my scatter gather algo nooo

inland radish Aug 29, 2024, 9:11 PM

#

flint idol ddd is just a gui for gdb

ahhh

#

Cool

flint idol Aug 29, 2024, 9:12 PM

#

it's better than any gui for gdb I've used, except for the one built in visual studio

#

I think I found a bug

#

if (((addr + bytesInPage) & ~(OBOS_PAGE_SIZE-1)) != (addr & ~(OBOS_PAGE_SIZE-1)))
{
    trunc = true;
    bytesInPage = pg_size - (addr % pg_size);
}```

#

here bytesInPage can be OBOS_PAGE_SIZE, thus causing a trunc to be true, causing fnuy stuff to happen

#

why would the AHCI driver read all zeroes though

#

this makes no sense

flint idol Aug 29, 2024, 9:25 PM

#

flint idol why would the AHCI driver read all zeroes though

ONLY ON QEMU TOO

#

to the qemu source I go...........

flint idol Aug 29, 2024, 9:54 PM

#

only one sector is read

#

rather that, or that part of the buffer was already filled from a previous

flint idol Aug 29, 2024, 10:22 PM

#

interesting...

#

despite there being quite a few reads with > 4 sectors

#

ide tracing only ever shows 4 sectors

#

unless that has something to do with scatter gather

#

I only ever see 1-4 sector reads

#

which seems to match what I see in my buffers

#

I was going to see what my third kernel said in the qemu log

#

but it crashes :(

#

turning off smp stops the crash

flint idol Aug 29, 2024, 10:31 PM

#

flint idol I was going to see what my third kernel said in the qemu log

completely different

#

even the command register is different

#

just found a bug in my old ahci drivre

flint idol Aug 29, 2024, 10:34 PM

#

flint idol even the command register is different

that might be from the bios

#

reads

#

the command registers match otherwise

#

weird...

#

the command never appears to get issued

#

on the IDE side

#

no error bits seem to be set

#

the only difference between the current implementation and my previous (third) kernel's impl is that one uses the PRDT with > 1 entry

#

any read with > 4 sectors seems to be dropped

#

ide_bus_exec_cmd IDE exec cmd: bus 0x55de5f6f6558; state 0x55de5f6f65e0; cmd 0x25```

#

usually when that comes in the IDE trace

#

there's something like this following it:

ide_dma_cb IDEState 0x55de5f6f65e0; sector_num=2363 n=1 cmd=DMA READ```

#

but for one command (probably the one with 0x139 sectors being requested) that doesn't happen

#

I will now make the ultimate qemu log

#

one with all tracing enabled

flint idol Aug 29, 2024, 11:22 PM

#

the buffer seems to be one sector short

#

so the ide thing probably drops the read

#

then ahci interrupts me saying the command is done

#

ahci_dma_prepare_buf ahci(0x55e01790a7e0)[0]: prepare buf limit=160256 prepared=159744```

#

the limit is that many bytes

#

but for some reason prepared is one sector short

#

memzero((void*)cmdTBL, sizeof(*cmdTBL));
for (uint16_t i = 0; i < data->physRegionCount; i++)
{
#if OBOS_ARCHITECTURE_BITS == 64
    if (!(HBA->cap & BIT(31)))
        OBOS_ASSERT(!(data->phys_regions[i].phys >> 32));
#endif
    memzero((void*)&cmdTBL->prdt_entry[i], sizeof(cmdTBL->prdt_entry[i]));
    AHCISetAddress(data->phys_regions[i].phys, cmdTBL->prdt_entry[i].dba);
    cmdTBL->prdt_entry[i].dw4 = ((data->phys_regions[i].sz - 1) & 0x3fffff) << 0;
    // cmdTBL->prdt_entry[i].i = (i == (data->physRegionCount - 1));
    // cmdTBL->prdt_entry[i].dw4 &= ~BIT(31);
    cmdTBL->prdt_entry[i].dw4 |= BIT(31);
}
cmdHeader->prdtl = data->physRegionCount;```
the problem probably lies here

flint idol Aug 29, 2024, 11:47 PM

#

flint idol ```c memzero((void*)cmdTBL, sizeof(*cmdTBL)); for (uint16_t i = 0; i < data->phy...

I think the bitmask was wrong

#

nvm

flint idol Aug 29, 2024, 11:51 PM

#

flint idol ``` ahci_dma_prepare_buf ahci(0x55e01790a7e0)[0]: prepare buf limit=160256 prepa...

limit is the amount of bytes needed

#

prepared is the amount of bytes in the PRDT

#

and after doing some math

#

||it's a bug with my scatter gather algo||

#

Ok

#

I really gotta stop writing code when I'm tired

#

if (bytesLeft <= pg_size && ((int64_t)size) > pg_size)
    bytesInPage -= initialBytesInPage;```

#

wtf is that even supposed to do

#

ahci_dma_prepare_buf ahci(0x55c700358450)[0]: prepare buf limit=160256 prepared=160256
ahci_cmd_done ahci(0x55c700358450)[0]: cmd done
ahci_trigger_irq ahci(0x55c700358450)[0]: trigger irq +DHRS (0x00000001); irqstat: 0x00000000 --> 0x00000001; effective: 0x00000001
ahci_check_irq ahci(0x55c700358450): check irq 0x00000000 --> 0x00000001
ahci_irq_raise ahci(0x55c700358450): raise irq```

#

that bug is fixed

#

dam slowfat really is slow

#

why is that part of the page cache not in the page cache!??!?!?!?!?!?!

#

fuck

#

this

#

shit

#

I am so close to rewriting ||the fat driver||

#

nvm I am just chronically skill issued

short mortar Aug 30, 2024, 12:33 AM

#

Operator precedence?

#

(switch to clang )

flint idol Aug 30, 2024, 12:34 AM

#

oh how fun

#

it now goes over the partition

short mortar Aug 30, 2024, 12:34 AM

#

flint idol ```c if (bytesLeft <= pg_size && ((int64_t)size) > pg_size) bytesInPage -= i...

(I had some bugs in code like that because of it)

flint idol Aug 30, 2024, 12:35 AM

#

flint idol ```c if (bytesLeft <= pg_size && ((int64_t)size) > pg_size) bytesInPage -= i...

I genuinely have no idea why this is here

#

time to use git blame

short mortar Aug 30, 2024, 12:35 AM

#

Alignment?

flint idol Aug 30, 2024, 12:35 AM

#

nope

#

initialBytesInPage is the amount of bytes in the first page

#

of the region

#

that code's been here since that function was implemented

#

and that line is run on the last page of the region

short mortar Aug 30, 2024, 12:37 AM

#

What does it even do that you're working with bytes in pages?

flint idol Aug 30, 2024, 12:37 AM

#

it's the thing that populates the PRDT

short mortar Aug 30, 2024, 12:38 AM

#

Oh

#

Yeah that's weird

flint idol Aug 30, 2024, 12:39 AM

#

the code seems to have been written around 11-12 PM

short mortar Aug 30, 2024, 12:39 AM

#

Though allowing your kernel to have a life of its own is an interesting design

flint idol Aug 30, 2024, 12:40 AM

#

flint idol the code seems to have been written around 11-12 PM

which might explain why I added it

flint idol Aug 30, 2024, 12:40 AM

#

short mortar Though allowing your kernel to have a life of its own is an interesting design

KEKW

short mortar Aug 30, 2024, 12:42 AM

#

Nah that's when you get all the "good ideas"

#

As in you can't use some of the C++ constructs in a clear state of mind

flint idol Aug 30, 2024, 12:46 AM

#

KEKW

#

my page cache is slow af

#

because on each resize

#

it allocates a new virtual memory block

#

so my idea is

#

to

#

just reverse the entire file

#

in the virtual address space

#

then allocate regions of that as needed

#

tbh my VMA is pretty bad

#

but now's not the time to rewrite it

flint idol Aug 30, 2024, 12:59 AM

#

flint idol just reverse the entire file

first I need to implement virtual address space reserving

#

should be pretty simple

#

make page nodes for pages

#

but don't back the pages by swap

#

or anything

#

or phys. memory

#

then on fault they're given a backing, or alternatively, you can just allocate the pages normally

#

I just pushed the incomplete writing code

#

before implementing reserving virtual address space in the VMA

#

that's almost done

flint idol Aug 30, 2024, 1:35 AM

#

done

#

well almost

#

actually I don't think I'll give the pages a backing on mapping

#

to aid with catching bugs

#

time to remake the page cache

#

it'll be the exact same

#

except

#

instead of resizing it

#

you check if someone already filled in the part of the page cache you want

#

if so then you're good

#

otherwise, you need to fill it in

#

then use it

#

to track that, I'll use a bitmap

#

actually...

#

nvm

#

I'll use a linked list

#

maybe...

#

I'll just query the page status

#

I'll abstract that behind VfsH_PageCacheGetEntry

#

void *VfsH_PageCacheGetEntry(pagecache* pc, void* vn, size_t offset, size_t size);

#

is good

#

I've made everything use that

#

I just need to implement it

#

Note to self: Implement cached async IO with the new page cache

flint idol Aug 30, 2024, 2:21 AM

#

omg this is possibly orders of magnitude faster

#

but I corrupt the disk image

#

like deep fry it

#

cook it

#

cleanse it

#

before testing the write

#

and after is too large to put here

flint idol Aug 30, 2024, 3:03 AM

#

basically the fat is corrupted

#

so are all directory entries

#

just minor corruption

flint idol Aug 30, 2024, 3:15 AM

#

flint idol so are all directory entries

this seems to happen when the FAT is flushed

#

TODOs for when I work on this tomorrow:

Fix the FAT code
Change the read code to follow the FAT cluster chain

#

just pushed the new page cache code

flint idol Aug 30, 2024, 10:39 AM

#

time to fix the FAT code

#

is it normal for sudo out of all things to lag

devout niche Aug 30, 2024, 10:59 AM

#

flint idol is it normal for `sudo` out of all things to lag

on linux it does seem slow

#

i wonder whether they deliberately insert a delay to prevent attempts at brute forcing the right password

#

they do the same with login

flint idol Aug 30, 2024, 11:00 AM

#

devout niche on linux it does seem slow

no like it's lagging, not being slow because I put the wrong password

real pecan Aug 30, 2024, 11:00 AM

#

flint idol is it normal for `sudo` out of all things to lag

No

vale nymph Aug 30, 2024, 11:02 AM

#

Even in astral sudo is pretty damn quick so I dont tjink thats normal

flint idol Aug 30, 2024, 11:03 AM

#

it's only ever happened today

#

guess something was being overloaded, idk

#

I think if I finish the fat driver today I can get to userspace by ~~the end of october~~ next friday

#

all I'd need is paging out to disk, signals, pipes, ttys, and syscalls

#

and some changes to the vmm

#

the FAT allocation code doesn't corrupt anything

vale nymph Aug 30, 2024, 11:11 AM

#

flint idol all I'd need is paging out to disk, signals, pipes, ttys, and syscalls

Join me on having working job control

#

All you need are signals a good tty system and sessions and process groups

flint idol Aug 30, 2024, 11:11 AM

#

sounds simple enough

#

I'll do that before I start working on syscalls

flint idol Aug 30, 2024, 11:12 AM

#

flint idol the FAT allocation code doesn't corrupt anything

nor does the code to copy the clusters

#

(when they're being reallocated)

#

I am now doing while(1) debugging

flint idol Aug 30, 2024, 11:14 AM

#

flint idol nor does the code to copy the clusters

nor does the code to free the clusters

#

there is only two things then:

the code to change the dirent filesize
the writing code

#

nor does the code to update the dirent

#

nor does the writing code......

#

it could be a pagecache bug

#

since I needed to disable caching on the partition to be able to use fsck on the partition while the kernel is running

#

I turned it off

inland radish Aug 30, 2024, 11:17 AM

#

then dont leave the kernel running

flint idol Aug 30, 2024, 11:17 AM

#

then the page cache doesn't get flushed

#

since I have no way to shutdown rn

#

only quitting qemu

flint idol Aug 30, 2024, 11:18 AM

#

flint idol since I have no way to shutdown rn

As in, no clean way

inland radish Aug 30, 2024, 11:18 AM

#

flint idol then the page cache doesn't get flushed

then add a clean way to flush it

#

some kind of ctrl-alt-delete style shortcut maybe

flint idol Aug 30, 2024, 11:18 AM

#

not my priority rn

inland radish Aug 30, 2024, 11:18 AM

#

if you dont have an interactive shell

vale nymph Aug 30, 2024, 11:19 AM

#

Or add the code to sync it every X seconds and just wait a bit

flint idol Aug 30, 2024, 11:19 AM

#

weird

#

nothing gets corrupted with the page cache on

#

actually nvm

#

I was testing that wrong

#

page cache on corrupts the FAT volume

#

so I need to fix the page cache

#

I'll log each uncached write, then the page cache dirty regions

#

and see how they differ

#

[ DEBUG ] writing bytes from 0x0000000000013200-0x00000000073c0000 (block size: 200)
[ DEBUG ] writing bytes from 0x0000000000026a00-0x0000000000040000 (block size: 200)
[ DEBUG ] writing bytes from 0x0000000000013200-0x00000000073c0000 (block size: 200)
[ DEBUG ] writing bytes from 0x0000000000026a00-0x0000000000040000 (block size: 200)
[ DEBUG ] writing bytes from 0x00000000025ff400-0x0000000000080000 (block size: 200)
[ DEBUG ] writing bytes from 0x0000000000000400-0x0000000000040000 (block size: 200)
[ DEBUG ] writing bytes from 0x0000000000013200-0x00000000073c0000 (block size: 200)
[ DEBUG ] writing bytes from 0x0000000000026a00-0x0000000000040000 (block size: 200)
[ DEBUG ] writing bytes from 0x00000000025ff400-0x000000007fd80000 (block size: 200)
[ DEBUG ] writing bytes from 0x0000000000000400-0x0000000000040000 (block size: 200)```

#

for cached

#

📎 uncache_writes.txt

#

that's for uncached

#

I will now examine these logs

#

I know what it is

#

it likely has something to do with alignment

#

actually nvm

#

the dirty regions being flushed could be extended

#

[ DEBUG ] flushing bytes from 0x00000000026ff000-0x0000000000080000 (block size: 200)

#

it's possible that I'm writing too much

#

nvm my logging is just bad

#

after making all the logs match

#

I get no difference between cached and uncached writes

#

in the log

#

obos has 30K loc now

#

don't know where all those came from

vale nymph Aug 30, 2024, 11:55 AM

#

flint idol obos has 30K loc now

More than astral what the fuck

flint idol Aug 30, 2024, 11:55 AM

#

how many does astral have

vale nymph Aug 30, 2024, 11:56 AM

#

Upper 20 something tjousand I believe

flint idol Aug 30, 2024, 11:56 AM

#

interesting

#

I wonder which part of my kernel has the most loc

#

probably arch/x86_64

vale nymph Aug 30, 2024, 11:57 AM

#

cloc had given me like 26 thousand lines of code, some thousand comments and some thousand blanks

flint idol Aug 30, 2024, 11:58 AM

#

I mean I have like a 31:4 ratio of code to comments

#

the kernel itself has 25k loc

#

drivers somehow amount to 5k loc

vale nymph Aug 30, 2024, 12:00 PM

#

Fun

#

Youve written more code in 4 months than me in a year and a few months

flint idol Aug 30, 2024, 12:01 PM

#

yeah I've been working on this all summer

#

since it's summer break, no school

#

means I get all day to osdev

#

so I use all day

flint idol Aug 30, 2024, 12:02 PM

#

vale nymph Youve written more code in 4 months than me in a year and a few months

you've written more ports than me than I have in one year and a few months

#

over all rewrites of OBOS

#

NONE of them have a single thing ported

#

closest thing was an mlibc port that didn't compile

vale nymph Aug 30, 2024, 12:07 PM

#

flint idol since it's summer break, no school

I miss summer breaks, last year I had school and a job and now I have uni and a job

vale nymph Aug 30, 2024, 12:07 PM

#

flint idol you've written more ports than me than I have in one year and a few months

Im lucky porting stuff is fun

#

If you need any help with porting stuff lmk

flint idol Aug 30, 2024, 12:07 PM

#

ok

#

(I was going to ping you when I needed help w/ ports regardless)

#

the disk images differ a lot

#

cached diffed with uncached is like completely different

#

the entire diff is 78 mib

#

which leaves ~2 mib that aren't different

#

which is 2048 sectors

#

which is coincidentally the entire length of the disk before the fat partition

flint idol Aug 30, 2024, 12:45 PM

#

instead of testing the pagecache with the fat driver

#

I will test it with some writes

#

on sda

#

the page cache keeps dirty regions after a flush

#

which it's not supposed to do

#

I've found the bug

#

*a bug

#

nvm

#

no pagecache bugs appear with my test

flint idol Aug 30, 2024, 1:38 PM

#

I changed back to testing the other thing

#

the fat driver

#

and even a 512-byte write (< than cluster size!!)

#

corrupts all

flint idol Aug 30, 2024, 1:54 PM

#

fixed that

#

that's weird...

#

the entire fat is reported to be free

#

fixed

#

yeah I'll be honest I have no idea what's happening

#

wtf

#

this line looks sus:

size_t new_cap = (dirty->fileoff + off + sz);

#

in the page cache dirty region code

#

(to create/extend a region)

#

Assertion failed in function populate_physical_regions. File: /home/oberrow/Code/obos/src/drivers/generic/ahci/interface.c, line 91. !found->reserved

#

I've gotten closer

#

actually nvm, it seems like it's just a bug

#

nvm

#

I wrote my assert wrong

#

nvm

#

I wrote it right

#

I fixed that

#

Assertion failed in function populate_physical_regions. File: /home/oberrow/Code/obos/src/drivers/generic/ahci/interface.c, line 90. !found->reserved

#

to be greeted by this

#

which is good

#

because it confirms my previous suspicions

#

I fixed it

#

as in

#

the fat corruption

#

basically the bug was

#

since most pages in the pagecache are marked as reserved

#

and aren't backed

#

the ahci prdt populating thing would populate parts of a PRDT with phys 0

#

when I would writeback parts of the page cache

#

and the reason parts of the dirty region were reserved was because the kernel extends page cache dirty regions if it sees there is already a region containing the offset

#

// size_t new_cap = (off + sz);
// TODO(oberrow): Does this need to be synchronized
// dirty->sz = new_cap;```

#

pagecache_dirty_region* dirty = VfsH_PCDirtyRegionLookup(pc, off);
if (dirty)
{
    if ((dirty->fileoff + off + sz) <= (dirty->fileoff + dirty->sz))
        return dirty; // we have space in this dirty region, return it
    // // not enough space, expand the region.
    // size_t new_cap = (off + sz);
    // // TODO(oberrow): Does this need to be synchronized
    // dirty->sz = new_cap;
    // OBOS_Debug("extending dirty region from offset 0x%016x to a size of 0x%016x bytes\n", dirty->fileoff, dirty->sz);
    // return dirty;
    asm("");
}```

flint idol Aug 30, 2024, 3:00 PM

#

flint idol and the reason parts of the dirty region were reserved was because the kernel ex...

and since the parts inbetween were never accessed, they were still marked as reserved

#

tbh it's hard to explain

#

it's reallllly inefficient though

#

~250 small writes

#

when flushing the page cache for the drive with the fat partition

#

after the write to the file

#

the solution is to combine contiguous regions

#

*a solution

#

I need to do that along with eliminating duplicates

#

ok now this is faster

#

I think the fat driver is almost done

#

paging out to disk is going to be pain to debug

#

I have a bug with the ahci driver

#

basically the PRDT count is > than the max amount of PRDTs

#

now I could fix that

#

or I could increase the amount of PRDTs

#

to some absurd amount

#

at the expense of using more memory

#

(guess which one I'm choosing)

#

there are still bugs with writing

#

/file.txt
  Contains a free cluster (38738). Assuming EOF.
/file.txt
  File size is 2097189 bytes, cluster chain length is 1920000 bytes.
  Truncating file to 1920000 bytes.```

#

but at least it's not the partition getting absolutely cooked

flint idol Aug 30, 2024, 5:01 PM

#

flint idol ``` /file.txt Contains a free cluster (38738). Assuming EOF. /file.txt File ...

this is very likely because some sector of the fat wasn't written

#

it looks like a bug with the actual fat code and not a bug in the vfs

#

since the cluster chain doesn't get cut off at a sector boundary

flint idol Aug 30, 2024, 5:13 PM

#

flint idol it looks like a bug with the actual fat code and not a bug in the vfs

I decided to make sure by doing uncached io on the disk

#

and it turns out to be a vfs bug

#

or it turns out there is no bug?

#

miracle debug ftw

#

Added a memory stat:

[  LOG  ] Currently at 27800 KiB of committed memory (4676 KiB pageable), 876 KiB paged out, 23124 KiB non-paged. and 37784 KiB uncommitted.```

#

the "uncommitted" stat

#

basically it just says how many memory is reserved

flint idol Aug 30, 2024, 5:34 PM

#

flint idol creating a file: expand the parent directory if needed make LFN entries make 8.3...

marker

#

anyway now I need to implement truncating files

#

luckily for me that's really easy

#

obos_status trunc_file(dev_desc desc, size_t blkCount)
{
    if (!desc)
        return OBOS_STATUS_INVALID_ARGUMENT;
    fat_dirent_cache* cache_entry = (fat_dirent_cache*)desc;
    fat_cache* cache = cache_entry->owner;
    if (cache_entry->data.filesize == blkCount)
        return OBOS_STATUS_SUCCESS;
    OBOS_ASSERT(cache_entry->data.filesize >= blkCount);
    if (blkCount > cache_entry->data.filesize)
        return OBOS_STATUS_INVALID_ARGUMENT;
    const size_t bytesPerCluster = (cache->bpb->sectorsPerCluster*cache->blkSize);
    uint32_t szClusters = ((cache_entry->data.filesize / bytesPerCluster) + ((cache_entry->data.filesize % bytesPerCluster) != 0));
    cache_entry->data.filesize = blkCount;
    uint32_t newSizeCls = ((cache_entry->data.filesize / bytesPerCluster) + ((cache_entry->data.filesize % bytesPerCluster) != 0));
    if (szClusters == newSizeCls)
        return OBOS_STATUS_SUCCESS;
    uint32_t cluster = cache_entry->data.first_cluster_low;
    if (cache->fatType == FAT32_VOLUME)
        cluster |= ((uint32_t)cache_entry->data.first_cluster_high << 16);
    TruncateClusters(cache, cluster, newSizeCls, szClusters);
    return OBOS_STATUS_SUCCESS;
}```

#

I don't need to worry about blkCount being zero, as that's handled by TruncateClusters

#

all I need to do is write the filesize

#

then I'm done

#

four more callbacks remain

#

mk_file, move_desc_to, remove_file, and set_file_perms

#

(in ascending order of difficulty)

#

set_file_perms is a 5 loc function

#

remove_file should be easy enough

#

shit

#

nvm

#

it isn't recursive

#

and can only remove empty directories

flint idol Aug 30, 2024, 6:34 PM

#

oh god....

#

when the summer started the os was at 7k loc

#

and now it's at ||30k||

white mulch Aug 30, 2024, 7:10 PM

#

flint idol and now it's at ||30k||

damn

#

my os was like 17k at the start of april and now its 28k so that's like 11k in 5 months

flint idol Aug 30, 2024, 8:34 PM

#

I'm back

#

and have implemented removing files

#

and will test it now

#

it worked first try

#

the vfs doesn't support removing files atm though

#

I'll need to implement that

#

truncating files works as well

#

I also need to implement updating the accessed date/time for dirents

#

but that's not top priority rn

#

time to implement moving files

#

I feel bad for future me implementing ext2

flint idol Aug 30, 2024, 11:09 PM

#

OBOS_ASSERT(false && "unimplemented");

#

wonder how many times I could find something similar in managarm/mlibc

#

9 times in managarm dam

vale nymph Aug 30, 2024, 11:15 PM

#

usually in assertos its __ensure(!"unimplemented")

flint idol Aug 30, 2024, 11:15 PM

#

I searched for unimplemented

vale nymph Aug 30, 2024, 11:15 PM

#

oh

flint idol Aug 30, 2024, 11:24 PM

#

flint idol time to implement moving files

that's done

#

BUT

#

ref_dirent is partially unimplemented

#

which is used in move_desc_to

#

and soon, mk_file

weary hound Aug 30, 2024, 11:28 PM

#

flint idol ```c OBOS_ASSERT(false && "unimplemented"); ```

you'd get better results with grepping for assert(!"

flint idol Aug 30, 2024, 11:29 PM

#

55 files

#

in managarm alone

#

mlibc has one file

#

with that search keyword

weary hound Aug 30, 2024, 11:30 PM

#

yeah in mlibc it's __ensure more likely

flint idol Aug 30, 2024, 11:30 PM

#

81 files

sharp pike Aug 30, 2024, 11:31 PM

#

weary hound Aug 30, 2024, 11:32 PM

#

weary hound you'd get better results with grepping for `assert(!"`

i mean in managarm not all uses of this pattern are todos

#

e.g. ```
drivers/block/ahci/src/port.cpp
240: assert(!"commandsInFlight < numCommandSlots, but submission queue was full");

#

since it's a bug because stuff got out of sync

flint idol Aug 31, 2024, 12:36 AM

#

moving files almost works

#

I just need to fix my short name generation

flint idol Aug 31, 2024, 12:55 AM

#

/file.txt.lol
  Bad short file name (filet~1l.ol).```

#

meanwhile in the debugger it looks fine

flint idol Aug 31, 2024, 1:22 AM

#

fixed

#

almost

#

fixed

#

well renaming (moving into the same directory)

#

works

#

but not moving

flint idol Aug 31, 2024, 1:42 AM

#

flint idol ```c OBOS_ASSERT(false && "unimplemented"); ```

this fails

#

so I guess I have to implement that

#

but I will do that in the morning

#

since I don't want to do that now

#

but after moving files is implemented

#

I will have to implement mk_file which will be easy since I'd already have most of the infrastructure

#

then I can merge the FAT driver

#

~~and add some helpers for the vfs~~

#

start on paging out to disk

flint idol Aug 31, 2024, 11:59 AM

#

I need to implement following cluster chains

#

it should be simple

flint idol Aug 31, 2024, 1:09 PM

#

that's done

#

I also made read follow the cluster chain

#

instead of assuming a contiguous chain

#

I will also make write do that

#

(when allocating)

#

and the probe function

#

reading works

#

or rather the new implementation works

flint idol Aug 31, 2024, 3:25 PM

#

flint idol and the probe function

I did that

flint idol Aug 31, 2024, 4:15 PM

#

I am currently reimplementing most of move_desc

#

since it was going to be hard to make it follow the cluster chain of the directory

#

and also I made some other changes that would've made it harder to simply change the function

flint idol Aug 31, 2024, 4:51 PM

#

finally done

#

I fucking hate the FAT filesystem

#

my god I can't wait to finish this

flint idol Aug 31, 2024, 9:41 PM

#

flint idol I am currently reimplementing most of `move_desc`

FINALLY done fixing that

#

at least renaming it (moving to the same directory) works

#

||```
Directory for ::/a_long_name_that_wont_fit

Fat problem while decoding 3 0
Streamcache allocation problem:: 3```||

#

 Volume Serial Number is 5AE3-00C6
Directory for ::/a_long_name_that_wont_fit

.            <DIR>     2024-08-27  11:26 
..           <DIR>     2024-08-27  11:26 
OTHER_~1 TXT        28 2024-08-27  11:27  other_file.txt
SUBDIR~1     <DIR>     2024-08-27  11:28  subdirectory
FI~1     LOL        37 2024-08-27  11:28  file.txt.lol```

#

fixed

flint idol Aug 31, 2024, 10:16 PM

#

there's a bug where an entry doesn't get removed properly

#

now I get:

[ ERROR ] FAT: Error following cluster chain: Unexpected free cluster. Aborting.```

#

oops

#

fixed that

#

now there is a bug with removing the old dirent

#

/file.txt.lol  and
/a_long_name_that_wont_fit/subdirectory/file.txt.lol
  share clusters.```

#

this only happens when moving a file to a parent directory

#

which is because for some reason, dirent_offset is at some random cluster

flint idol Aug 31, 2024, 10:55 PM

#

I fix that

#

and suddenly the dirent name is getting zero?

#

ok it was because I was corrupting memory

#

because of a buffer overflow

#

because I was trying to write a cluster into a sector-sized buffer

flint idol Aug 31, 2024, 10:57 PM

#

flint idol ``` /file.txt.lol and /a_long_name_that_wont_fit/subdirectory/file.txt.lol sh...

aaand this is fixed

#

creating files will be easy af since I already have like

#

everything

#

to do that

flint idol Sep 1, 2024, 12:21 AM

#

I have implemented mk_file

#

#

file.vn->mount_point->fs_driver->driver->header.ftable.mk_file(&new_desc, UINTPTR_MAX, file.vn->mount_point->device, "file.txt", FILE_TYPE_REGULAR_FILE);

#

I can create files

#

rather, I can in the driver

#

the vfs doesn't support such a thing

#

as of now

#

when I work on syscalls I'll add support for that in the vfs

#

https://github.com/OBOS-dev/obos/pull/24

GitHub

FAT Driver by oberrow · Pull Request #24 · OBOS-dev/obos

Implements a FAT driver, and also fixes many bugs in the VFS (mostly related to the pagecache).

#

Can finally merge the fat driver

#

guess I can merge it

#

next up is paging out to disk

#

which I will start tomorrow

#

If someone could test obos on real hw (it won't do any writes to disk assuming you're on master) that would be cool

#

if you're paranoid that it will you can probably comment out some lines under src/oboskrnl/vfs/fd.c related to flushing and writing

flint idol Sep 1, 2024, 1:07 AM

#

flint idol If someone could test obos on real hw (it won't do any writes to disk assuming y...

also send a pic if you do

#

(of the monitor)

#

and ofc I have build instructions

#

here

#

so swapping out to disk should be trivial

#

I'm thinking I can have a freelist of pages stored on disk

#

to implement swap_resv and swap_free

#

and the file/partition will be represented using a vnode

#

and as long as driver devs (me) don't do anything stupid with the OBOS_PAGEABLE_FUNCTION macro

#

all will be good

#

I will have to implement moving memory from one swap device to another

#

but that'll be easy

#

just lock the context

#

brain fart

#

allocate all the neccessary space in the new swap device

#

copy the pages from the current swap device into the new one

#

then free the pages in the current swap device

#

it won't be fast, nor pretty

#

but it will only be done once in the kernel

#

if all goes smoothly this takes a day or two

#

if all does not go smoothly, then it will take anywhere from a week to months

#

or a rewrite meme

flint idol Sep 1, 2024, 11:58 AM

#

I will most likely not switch from in-ram swap to disk swap in the kernel

#

and have a syscall to do so

#

I will still test it ofc

flint idol Sep 1, 2024, 1:02 PM

#

I can initialize partitions as "swap" now

#

I just need to implement the code to manipulate the partitions

#

which should be pretty easy

#

swap_resv simply gets a node from the freelist to use as the area in swap for the page

#

swap_free adds the node to the freelist

#

swap_write and swap_read literally just ask the driver (nicely) to write/read the node passed from the kernel

#

and on shutdown, all it will do is reset the free list to span the entire partition once again

#

and it will clear flags.dirty

#

if there's any confidential memory left in the swap after

#

that'll be tough

#

for the person owning said data

#

anyway that's enough yapping, I need to get started

flint idol Sep 1, 2024, 2:12 PM

#

after many work

#

I should be able to reserve swap space in the file

#

writing will be trivial

#

just a call to the driver's write thing

#

shit

#

it takes in a phys. address

#

nvm that's the least of my worries

#

if blkSize > page size

#

I'll still need to be writing blkSize

#

but phys is page size length

#

maybe that's unrealistic

#

OBOS_ASSERT((blkSize <= OBOS_PAGE_SIZE) && "Block size being larger than the page size is unimplemented.");

#

Fixed.

#

or even better:

if (blkSize > OBOS_PAGE_SIZE)
    return OBOS_STATUS_UNIMPLEMENTED;

flint idol Sep 1, 2024, 3:50 PM

#

I should be able to page in and out from disk!

flint idol Sep 1, 2024, 3:50 PM

#

flint idol I will have to implement moving memory from one swap device to another

I just need to implement this

#

and swap_free

#

which I will implement in a couple hours, gtg now

flint idol Sep 1, 2024, 7:56 PM

#

BACK

#

Time to finish this

#

my friend found a computer with an amd athlon 64

#

and I'm trying to get him to test obos on it

flint idol Sep 1, 2024, 9:05 PM

#

flint idol Time to finish this

swap_free is implemented

flint idol Sep 1, 2024, 9:12 PM

#

flint idol I will have to implement moving memory from one swap device to another

done

#

now if all goes as planned

#

this should work

#

(I highly doubt all will go as planned)

#

recursive lock

#

shit

#

the ahci driver uses the vmm

#

obos_status status = Mm_VirtualMemoryProtect(CoreS_GetCPULocalPtr()->currentContext, (void*)(base - base % OBOS_PAGE_SIZE), size, OBOS_PROTECTION_SAME_AS_BEFORE|OBOS_PROTECTION_CACHE_DISABLE, false /* non-pageable */);

#

I can work around this by checking if it were non-pageable

#

and if it is, then I can skip the call

#

yeah that did it

#

I now will put my trust into my functions

#

and hope I don't corrupt the disk image

#

whoops

#

nvm hidden symbol

#

in a driver

#

holy shit

#

it works

#

idk if this is fake or not

#

Assertion failed in function Core_SemaphoreAcquire. File: /home/oberrow/Code/obos/src/oboskrnl/locks/semaphore.c, line 26. Core_GetIrql() <= IRQL_DISPATCH

#

when testing it further

#

the vma is calling swap out at a high irql

#

which thus calls the swap interface's resv thing

#

which calls read_sync

#

which attempts to acquire a semaphore

flint idol Sep 1, 2024, 9:26 PM

#

flint idol the vma is calling swap out at a high irql

don't even know why I made it be that high

#

yeah it works at dispatch, idk why I didn't originally do that

#

so yeah I can swap out to disk now

#

uint8_t* buf = OBOS_KernelAllocator->Allocate(OBOS_KernelAllocator, 0x10000, nullptr);
memset(buf, 0x1d, 0x10000);
page what = {.addr=(uintptr_t)buf};
page* found = RB_FIND(page_tree, &Mm_KernelContext.pages, &what);
Mm_SwapOut(found);
OBOS_ASSERT(memcmp_b(buf, 0x1d, 0x10000));
OBOS_KernelAllocator->Free(OBOS_KernelAllocator, buf, 0x10000);```

#

I have no idea how to test it

#

that was all I could come up with

#

~~I'll just move uACPI init~~

#

that actually might work...

#

and by work I mean not crash in pnp

#

it is very slow

#

but it initializes

#

my swap passes uStressTest

#

NEXT!

#

merging

#

everything from signals to (but not including) ports will be in a userspace-work branch

#

I'm genuinely surprised it passed uStressTest first try

#

time to implement some of these

#

it seems like kill simply adds a pending signal to a thread

#

and the pending signal list is queried on each (user-mode) context switch to a thread

#

calling a signal seems fairly straightford

#

wait where do I put the "signal trampoline"

vale nymph Sep 1, 2024, 9:55 PM

#

flint idol wait where do I put the "signal trampoline"

for astral its in mlibc, I pass it to the kernel in the sigaction struct (restorer field) and then that gets added to the signal stack when the signal gets called (such that when the ret instruction in the signal code returns it goes straight to the trampoline which then calls sigreturn)

flint idol Sep 1, 2024, 9:55 PM

#

ah ty

vale nymph Sep 1, 2024, 9:56 PM

#

its very inspired by how linux does it

#

in linux it can live somewhere in the vdso tho https://man7.org/linux/man-pages/man7/vdso.7.html

flint idol Sep 1, 2024, 9:58 PM

#

I saw that in the man page for sigreturn

#

now I have a very difficult decision to make

#

and that is ||which subdirectory should I put signals under||

#

I'm thinking to put it in the kernel root or under scheduler/

#

I'll just put it under the kernel src root

#

I'm looking at https://github.com/managarm/mlibc/blob/c8974ae9c2e2228842921c543e7435f045e62ba3/sysdeps/managarm/generic/signals.cpp to see what signal functions I'll need

#

seems pretty trivial

#

I can man the functions to see what they do

#

(btw, I am implementing signals solely for userspace, they will barely be used in the kernel)

flint idol Sep 1, 2024, 10:04 PM

#

flint idol I'm looking at <https://github.com/managarm/mlibc/blob/c8974ae9c2e2228842921c543...

I'll ofc have some other kernel-only functions

#

to execute signal handlers

#

btw if I have mlibc, can I get a c++ stdlib without much effort

#

like assuming I implemented a lot of sysdeps

#

NO WAY

#

they finally added guidance on how to port

#

turned out I can just switch to tty7 or 6 (idr which one is X)

#

it's tty7

flint idol Sep 1, 2024, 11:43 PM

#

after doing nothing productive, I will finally start

#

enum {
    SIGHUP     , SIGINT,       SIGQUIT,     SIGILL,       SIGTRAP,
    SIGABRT    , SIGBUS,       SIGFPE,      SIGKILL,      SIGUSR1,
    SIGSEGV    , SIGUSR2,      SIGPIPE,     SIGALRM,      SIGTERM,
    SIGSTKFLT  , SIGCHLD,      SIGCONT,     SIGSTOP,      SIGTSTP,
    SIGTTIN    , SIGTTOU,      SIGURG,      SIGXCPU,      SIGXFSZ,
    SIGVTALRM  , SIGPROF,      SIGWINCH,    SIGIO,       SIGPWR,
    SIGSYS     , SIGRTMIN,     SIGRTMAX = 64,
};```

#

I have shamelessly stolen kill -l's signums

#

so I'm thinking that I just have an array of 64 "signal headers" in each thread object

#

and a bitmap of which ones are pending

#

then to find one to dispatch, I can just __builtin_ctz the bitmask

#

iirc when a signal handler is running that signal is masked until sigreturn is called?

#

probably says that behaviour in signal.7

#

flint idol Sep 1, 2024, 11:53 PM

#

flint idol and a bitmap of which ones are pending

basically:
__buitlin_ctz(pending | ~masked)

#

I think?

rapid pulsar Sep 1, 2024, 11:53 PM

#

flint idol ```c enum { SIGHUP , SIGINT, SIGQUIT, SIGILL, SIGTRAP, ...

You probably shouldn't use 0 as a signal number.

flint idol Sep 1, 2024, 11:53 PM

#

that was an artifact from when I was making the kill -l list an enum

#

but I fixed it

#

// For internal use.
typedef struct signal_desc {
    uintptr_t entry;
} signal_desc;
typedef struct signal_header {
    signal_desc signals[64];
    // NOTE: To get the first signal to dispatch, use __buitlin_ctz(pending | ~masked)
    uint64_t    pending;
    uint64_t    mask;
} signal_header;```

#

that should be good

#

there will be a signal_header in a thread context

flint idol Sep 2, 2024, 12:02 AM

#

flint idol ```c // For internal use. typedef struct signal_desc { uintptr_t entry; } si...

I think I'll make signal_desc waitable

#

then I won't have to implement sys_sigsuspend

#

in the kernel

#

and just emulate it in mlibc

flint idol Sep 2, 2024, 12:12 AM

#

flint idol ```c // For internal use. typedef struct signal_desc { uintptr_t entry; } si...

signal_desc will also contain a sigaction

#

wait

#

that struct is basically what a sigaction is

flint idol Sep 2, 2024, 12:29 AM

#

what does the third parameter of sigaction (in the struct "sigaction") do?

void(*sigaction)(int signum, siginfo_t* info, void* unknown);```

#

I think it passes info->val.ptr

#

but idk

rapid pulsar Sep 2, 2024, 12:49 AM

#

Do you mean sa_sigaction, the field member, that takes a void*?

flint idol Sep 2, 2024, 12:49 AM

#

yes

rapid pulsar Sep 2, 2024, 12:50 AM

#

Undocumented
Before the introduction of SA_SIGINFO, it was also possible to get some additional information about the signal. This was done by providing an sa_handler signal handler with a second argument of type struct sigcontext, which is the same structure as the one that is passed in the uc_mcontext field of the ucontext structure that is passed (via a pointer) in the third argument of the sa_sigaction handler. See the relevant Linux kernel sources for details. This use is obsolete now.

flint idol Sep 2, 2024, 12:50 AM

#

I see

#

so I should just be able to pass NULL

rapid pulsar Sep 2, 2024, 12:51 AM

#

If SA_SIGINFO is specified in sa_flags, then sa_sigaction (instead of sa_handler) specifies the signal-handling function for signum. This function receives three arguments, as described below.

#

   ucontext
          This is a pointer to a ucontext_t structure, cast to void *.  The structure pointed to by this field contains signal context information that was saved on the user-space stack by the kernel; for details, see sigreturn(2).  Further information about the ucontext_t structure can be found in getcon‐
          text(3) and signal(7).  Commonly, the handler function doesn't make any use of the third argument.

flint idol Sep 2, 2024, 12:52 AM

#

ah

#

what man page is this, signal.7?

rapid pulsar Sep 2, 2024, 12:52 AM

#

sigaction(2)

flint idol Sep 2, 2024, 12:52 AM

#

ah

rapid pulsar Sep 2, 2024, 12:53 AM

#

But you can also just reference POSIX: https://pubs.opengroup.org/onlinepubs/9699919799/functions/sigaction.html

rapid pulsar Sep 2, 2024, 12:58 AM

#

flint idol then I won't have to implement `sys_sigsuspend`

Note that sigsuspend should atomically set the mask and then wait. Can you do that without a signal being delivered after you set the mask but before you wait? (I made this mistake myself!)

flint idol Sep 2, 2024, 12:59 AM

#

maybe not...

#

at least not in usermode

#

guess I'll implement that as a syscall

rapid pulsar Sep 2, 2024, 1:00 AM

#

Usual usecase for sigsuspend is to block signals in a critical section, then sigsuspend to allow them through again, so if one was marked pending during your critical section and you naively set the mask from userspace you can end up with that signal immediately unblocked and its handler called, then waiting indefinitely for another signal immediately after

#

You are inspiring me to finish the rest of my signal implementation, in particular the SIGINFO stuff. Might have to actually un-archive my repo :)

flint idol Sep 2, 2024, 1:01 AM

#

rapid pulsar Usual usecase for sigsuspend is to block signals in a critical section, then `si...

in fact, I think I've had that exact problem in my kernel elsewhere

#

with events

#

in waitonobject

#

it could run at IRQL_PASSIVE

flint idol Sep 2, 2024, 1:02 AM

#

flint idol it could run at IRQL_PASSIVE

*DISPATCH

#

but would be lowered back right before the thread is blocked

#

so if there was a DPC that set that event

#

that was run by LowerIrql

#OBOS (not vibecoded)