#The Flopperating System (tfos) - x86 32 bit Hobby OS

1 messages · Page 5 of 1

bold grove
#

Do you think I could keep this “address space” local to the filesystem

#

Because it kinda conflicts with how address spaces work in mine

small hound
#

uhhh wdym

#

you can name it something different sure

bold grove
#

Imma just name it file_data_space_t

bold grove
# 

struct page_cache {
    struct Page *free_list;
    spinlock_t lock;
} page_cache;

void page_cache_init(void) {
    page_cache.free_list = NULL;
    static spinlock_t initializer = SPINLOCK_INIT;
    page_cache.lock = initializer;
    spinlock_init(&page_cache.lock);
}

void *page_cache_alloc(void) {
    spinlock(&page_cache.lock);
    if (page_cache.free_list) {
        struct Page *p = page_cache.free_list;
        page_cache.free_list = p->next;
        p->is_free = 0;
        spinlock_unlock(&page_cache.lock, true);
        return (void*)p->address;
    }
    spinlock_unlock(&page_cache.lock, true);
    void *page = pmm_alloc_page();
    return page;
}

void page_cache_free(void *addr) {
    if (!addr) return;
    struct Page *p = phys_to_page_index((uintptr_t)addr);
    if (!p) return;
    p->is_free = 1;
    spinlock(&page_cache.lock);
    p->next = page_cache.free_list;
    page_cache.free_list = p;
    spinlock_unlock(&page_cache.lock, true);
}
#

I make a lock for each page cache

small hound
bold grove
#

Okay I’ll look into it

violet anchor
#

education schmeducation

bold grove
violet anchor
#

educational is relative, what are you wanting to teach? parts of my os are educational, but not the kernel

bold grove
#

I wanna teach mainly the process

#

And the issues you encounter

#

Like I am right now lol

#

Making my file system not dogshit

violet anchor
#

it's more the case of "use the os to learn how to mess around with modern hardware at the bare metal level with no restrictions" not "learn how to write a kernel"... I'm not Tannenbaum

bold grove
#

Yeah same

#

I’m not doing the whole

#

“OSdev tutorial step 1: make GDT 🤓🤓”

violet anchor
#

my os exposes a userland at ring 0 in a language that's very easy to learn, that's unique and useful in an educational setting as you can instruct how things like drivers work without having to also know C

bold grove
#

Oh that’s interesting

#

There’s definitely lots to learn from that

violet anchor
#

that's how mine is "educational", so yeah education doesn't always posit perfect kernel code

bold grove
#

Yeah nah my kernel is dogshit

violet anchor
#

you just gotta specify what you expect someone to learn and from what

bold grove
#

I guess I want people to learn of the process

#

And how a normal person might go about this

violet anchor
#

mine sure isn't perfect it's got 16 years of legacy cruft in parts

bold grove
#

Mine is about a year of pain

#

But yeah I guess I can’t be educational truly

#

I think I’m just going for

violet anchor
#

are we really "normal people"? normal people don't write OSes kekeke

bold grove
#

And I wasted my time writing a really bad one

#

😔

#

Well I don’t have physical storage @small hound

#

So a page cache is useless

violet anchor
#

if it's useful to you it's not wasted

#

I write my os because I will use it

bold grove
#

I don’t use mine at all besides for fucking around on my old pentium laptop

#

But I haven’t booted it on there in months

violet anchor
#

just have a user in mind and write for that user, that user being you is just as legit as any else

bold grove
#

Yeah true

#

I’m more concerned about the internals

#

At the current moment

violet anchor
#

does it still boot there?

bold grove
#

I’m not sure

#

Most likely yeah

violet anchor
#

I find if I go more than a few days now without retesting I might break something, but I'm now past dicking with early boot

bold grove
#

The only thing that prevented booting a few times was the memory map being dogshit

small hound
bold grove
#

So the page cache would just live in volatile memory then?

small hound
#

it's not just for reducing IO operations

violet anchor
#

mmap for files is very much a unix thing isnt it? what's the windows equivalent?

small hound
violet anchor
#

I remember it having one

bold grove
#

But yeah

small hound
#

sure yea

bold grove
#

So the page cache should live in the ram

#

Okay

small hound
bold grove
#

Should that be its own address space too

small hound
bold grove
violet anchor
#

that's the one

small hound
bold grove
#

Yeah

#

I’m reading this paper rn

#

But all of them talk about physical storage

violet anchor
#

I prefer the NT syscall naming, the people who wrote unix and POSIX are scared of variables and functions over 6 chars long.

ENOMEM, EFILE, EIO... like wtf. why did nobody have proper descriptive const names

bold grove
#

They’re all shit

small hound
bold grove
#

For naming at least

bold grove
small hound
bold grove
#

That’s why globl is the way it is in some assemblers

violet anchor
#

and libc is just as bad... stricmp vs strcasecmp etc, make up your minds

bold grove
#

@small hound demand paging is for the user space right

small hound
#

no you can use it anywhere

#

or not anywhere, but you can use it in the kernel too

violet anchor
small hound
#

you probably don't want to demand page an interrupt stack

bold grove
#

Yeah

#

Def not

violet anchor
#

btw am I the only one when I see union in C code my mind sees onion 🧅

bold grove
#

Yes me too fr fr

violet anchor
#

oh god my eyes

#

twitch twitch

small hound
#

wowzers

bold grove
#

Yeah wtf

violet anchor
#

goes back to college days

small hound
#

aren't you like double that age now

#

wowzers

violet anchor
#

a friend of a friend thought I would take over the world

#

so he started calling me brain and calling my friend pinky

#

I owned the name, narf. my friend hated being called pinky and that didn't stick for him

small hound
#

narf brain and pinky name a more iconic trio

violet anchor
#

yeah I'm 43, so I've been known as brain longer than not

#

wait discord has a bug for me every gif is flickering rapidly like fit inducing flicker

bold grove
bold grove
#

I could create a page cache by pretending part of my memory is a block device

small hound
#

uhhhhhhhhhhhhhhhhh

bold grove
#

New pages are added to the page cache

small hound
#

noooooo

#

keep reading

#

you'll get it eventually

bold grove
#

The page cache is the main disk cache used by the Linux kernel. In most cases, the kernel refers to the page cache when reading from or writing to disk. New pages are added to the page cache to satisfy User Mode processes’s read requests. If the page is not already in the cache, a new entry is added to the cache and filled with the data read from the disk. If there is enough free memory, the page is kept in the cache for an indefinite period of time and can then be reused by other processes without accessing the disk.

small hound
#

it doesnt say anything about the "memory being a block device"

bold grove
#

Yeah I know but technically you would need a “disk”

#

But I don’t have that

#

So it will have to just store the page cache in ram

small hound
#

yes the page cache does live in ram

#

did it say anything about it living on disk

bold grove
#

I was confused, I thought it was stored on the disk

#

But in reality

#

That’s not the case lol

small hound
#

there is the block cache, where fast disks are used to cache data from slow disks, but that's another thing

violet anchor
#

I'm considering adding a similar write through cache to my os, it's in my roadmap

bold grove
#

My road map is

small hound
violet anchor
#

right now most reads hit the disk only thing that doesn't is tree walk to dirs we've seen before

#

the tree is cached to ram and written through and the file metadata within those dirs

violet anchor
#

it's mostly straightforward stuff now

bold grove
#

Nice nice

bold grove
violet anchor
#

a lot of things on that list are boring to me to implement, like I'd find an existing radix tree

#

horses for courses, I guess you do whatever bits interest you

#

I used someone else's hash map rather than reinvent that wheel

#

also I have two levels of allocator, the lowest level allocator is primitive O(n) allocate and free, but you aren't supposed to use that for everything, you use that to allocate long lived blocks and then initialise a separate buddy allocator within the block, when you're done with all allocations in that block you can just tear down the underlying one. kinda like how a process gets its own page table in a traditional os, but in my case not paging, as it's a purely interpreted environment

bold grove
bold grove
#

Below page size -> slab
Above page size -> buddy

#

Actually technically not

#

My physical allocator uses a buddy

#

My kernel heap uses a free list for allocations above page size

bold grove
#

Working on my implementation

paper cloak
#

the inherent problem with "educational OSes" is you either have to say this or just dont make an educational thing at all

#

all educational OSes are toys

#

which is the point

small hound
violet anchor
#

Linux kernel is a mess

#

it's a combination of a thousand independent silos of code with different levels of stability and code hygiene

#

each written by different teams or people

#

this is a result of scale, which we don't see at our OSes sizes

#

kinda like if windows kernel came with a thousand third party drivers

#

it's stable but probably the worst thing to try and learn from

paper cloak
violet anchor
#

yes it was the opposite

#

it was rebellion against minix never wanting to grow beyond educational

#

I think that's what gummi meant

#

forgot the /s tag

paper cloak
#

but probably because they have a small team of core contributors

small hound
#

meanwhile linux ranges between "wtf" to "pretty good"

paper cloak
#

linux is extensively documented at least

#

which is nice

#

for example RCU documentation is very nice

#

I guess it depends on the subsystem's author

bold grove
#

@small hound can you explain how to do a tlb shootdown for all CPUs

#

I don’t need code but I remember we talked about it

small hound
#

I thought you already implemented it?

bold grove
#

For many CPUs?

#

I thought it would be different

small hound
#

the smarter way is to batch this and shootdown a list of pages rather than one page at a time

bold grove
#

And create some thread to do it for me

#

Like a thread to do the shoot downs for me

small hound
#

why would you want to spawn a whole thread for this and use a global list

#

if you have DPCs it would be more optimal to use those instead of having a dedicated thread

#

a global list is pretty unnecessary because you'll often have scenarios where you know some core can't have a given page mapped

#

so you just skip that core

#

you'd do core local lists

bold grove
#

I perform a tlb shootdown every time I map/unmap or otherwise access page tables right?

#

And I’m sure my vmm_protect will need that too

small hound
# bold grove I perform a tlb shootdown every time I map/unmap or otherwise access page tables...

you don't need to shootdown if you're mapping a brand new page that you can guarantee will not have a mapping anywhere else (you are not overwriting an existing mapping)

for example if you have your vaddress space allocator and it creates a new page that has never been mapped, the shootdown isn't necessary

or if you're in your early kernel boot, and youre mapping the kernel, no one will have the mappings so there is no need to do the shootdown

similarly, if you can guarantee that some page will not be mapped on any other core, such as with a userspace process that has only ran on one core, when you unmap the pages for it, there is no need to do the tlb shootdown

bold grove
small hound
bold grove
#

In the pt_level function I’ll do that

#

And set some condition to be true

#

And only tlb shootdown

#

If that condition was set

#

So psuedo code would be

#

If (page_was_mapped)

tlb_shootdown();
Return;

Else return;

small hound
#

and then you can also check if the page is even mapped on the other cores if you're changing userspace pages and whatnot

#

since if you have some single threaded process that has only ran on one core you can guarantee that its pages will not have mappings on the other cores

#

💥

#

have a blast 😃

bold grove
bold grove
#

Aight boys

#

Working on the page cache now too

small hound
#

do take great care when doing this

#

you may have realized that there is a major potential source of fs corruption

bold grove
small hound
bold grove
#

Without an actual syscall

small hound
#

that's crazy

bold grove
#

Definitely not

small hound
#

no that's not an issue

#

you'd only mark em as user accessible upon the syscall

#

so if they are accessed before that you just get a page fault

#

so there's no issue

#

user process just gets terminated

bold grove
#

Wait no

#

Here

small hound
# bold grove If any of the pages used by a file system get accessed by the user

ok so the source of corruption I am referring to is the block device page cache

you know how you have stuff like /dev/sda

well, what if userspace modifies some arbitrary page in there and that page happens to be filesystem metadata information

then, the rest of the in memory representation of the filesystem would become corrupt

#

so how do you resolve this predicament

#

there are many different ways to solve this, or you can just pretend like this won't happen and accept the corruption lmao

bold grove
#

And access control

#

That’s what my guess what be

small hound
#

so that doesn't really do anything

bold grove
#

Access control?

#

What about something like journaling in ext 2

small hound
# bold grove Access control?

what does that mean though? there are both data and metadata pages in the block device page cache and it's safe to modify and write back data pages because those don't change the filesystem structure

small hound
bold grove
#

Only allowing root to access /dev/

#

Wouldn’t that prevent the issue

#

So the user can’t change the state of the disk to be different from the metadata state

small hound
small hound
bold grove
small hound
#

remember how sector sizes are typically 512

#

and also that doesn't solve the issue of metadata corruption lol

bold grove
small hound
#

it just kinda avoids the problem but prevents any modification of the block device page cache which is suboptimal

#

but sure you can just do that and come back later to actually try and resolve this predicament

#

😃

bold grove
small hound
#

bsd 😃

#

spoiler: ||they don't prevent it||

bold grove
#

Really

#

Wtf

#

Then I have no clue

bold grove
#

I’m still thinking about this issue lol

small hound
small hound
#

try focusing on actually getting started on the page cache first

#

just be aware of that problem

#

i can tell u about a few potential ways of resolving this later

bold grove
#

Dont allow bizarre syscalls

small hound
#

syscalls do get sanitized anyways so you don't pass in a buffer at 0x0 or something lol

bold grove
#

Prevent situations that could corrupt it in the first place

small hound
bold grove
#

I see

#

Idk then

small hound
#

you're on kind of the right track

#

it doesn't really happen at the syscall level but you're kinda on the right track

bold grove
#

Sanitize calls to tmpfs_write read etc?

small hound
bold grove
#

Okay

small hound
# bold grove Okay

and to answer your question about how linux and the BSD's handle this, they don't, the filesystem just becomes corrupt lol (in memory fs doesn't match what's on disk)

#

I know it's like this on linux and I believe it's like that on the BSDs

#

you can double check for the BSDs

#

that's why stuff like fsck will yell at you if you try to run it on a mounted filesystem

#

it still lets it happen though

small hound
paper cloak
paper cloak
#

Also you mean like block cache/buffer cache right?

paper cloak
bold grove
small hound
small hound
paper cloak
#

Kinda weird

#

There's typically a block cache too tho

paper cloak
small hound
paper cloak
#

I'd store FS metadata in a separate cache (block/buffer)

paper cloak
paper cloak
small hound
# paper cloak I'd store FS metadata in a separate cache (block/buffer)

yea but if you write to the block device page cache you need some way to identify if you're writing to metadata or data blocks to determine if you should actually write it to disk or not

for example if you use the block device page cache and modify the dirents of an inode, then the corresponding dirent cache would become invalid and thus corrupt since it doesn't match what's on disk

#

mildly problematic

paper cloak
small hound
#

give it a shot

paper cloak
#

I don't think it's such an issue as you make it to be

#

You can just let the user corrupt it if they want to

small hound
#

💔

bold grove
viscid zephyr
#

It's not just about users modifying it, it's also an issue with the system modifying it

#

If you reuse your page cache for your fs metadata caching, and you have a page of metadata that contains a few sectors in common with a page of data, and they get out of sync, you'll have a bad time

bold grove
#

Maybe I don’t and I’m dumb but I hope i do

bold grove
#

Okay I’m kinda almost done with my page cache implementation

#

@small hound do you have any decent page cache implementation I can look at

small hound
#

I haven't looked at freebsd's

#

linux and their whimsical naming strikes again

#

bread 🍞

bold grove
bold grove
thorn dust
small hound
#

amen

bold grove
#

I’m having major skill issues rn

bold grove
#

Okay I might need to make the rest of the operating system thread safe

bold grove
#

@viscid zephyr sorry to ping but i just need help. is it okay to preempt with a timer like the apic?

bold grove
thorn dust
#

but beware a context switch being tied inherently to an interrupt

bold grove
thorn dust
#

the proper way to implement context switching is as glorified setjmp()/longjmp() sort of thing

#

say thread B is running in userspace. a timer interrupt arrives and you determine that its timeslice is over so you have to switch. here is how the stack might look at the point of switching
someUserlandFunction() -> [interrupt frame] -> handleTimerInterrupt() -> switchThread()

your switchThread() ought to save the state of the preempted thread, such that when control is returned to it by another switchThread(), control returns to:
someUserlandFunction() -> [interrupt frame] -> handleTimerInterrupt()
as in, to the line immediately after the one that called switchThread()

#

suppose handleTimerInterrupt() is like:

{
  /* ... */
  if (timesliceExpired)
    switchThread();
  /* control resumes here */
  return;
}

at the point it reaches the end of handleTimerInterrupt(), your interrupt handling routine in asm will restore all the registers and return to userland. that's how you get back into userland after switching back to that thread

paper cloak
thorn dust
#

it follows as well that you can just as well call switchThread() from any context, interrupt or not

bold grove
bold grove
#

i might have 80 questions soon

small hound
#

how did you write a page cache so quickly btw

bold grove
#

i made an lru list

small hound
#

how

bold grove
#

and stuff

bold grove
#

ill show tests later

small hound
#

no I mean "how" as in "this sounds like you didn't actually implement a page cache because these things are far more nuanced and complex than anything that you could just 'sit down, get the jist of, and write in one day'"

#

did you copy an existing impl or something

bold grove
#

no. i just read some papers

#

and made my thing

#

i dont think its 100% correct yet

#

im still working on it

small hound
#

do you have asynchronous writeback

#

or is it writethrough

bold grove
#

im working on Cow

small hound
#

are you implementing a filesystem with cow

bold grove
#

yeah is that bad

small hound
#

no but what does "working on cow" mean

#

is that what you meant

small hound
bold grove
bold grove
#

Idk

#

I just wanna support snapshots

small hound
#

what do you mean "you don't know"

small hound
bold grove
#

Yes

#

I know it’s used in zfs and btrfs

#

So I’m studying those

#

And I know sql uses it

small hound
bold grove
#

Yeah probably

#

I’m also just making a vfs and tmpfs

small hound
# bold grove I just wanna support snapshots

also snapshots are very nontrivial and quite complex to implement and if you have only written an incorrect ext2 driver up to this point then I advise not doing this and first writing a functional driver for an on disk filesystem

#

just a hunch

bold grove
#

@small hound I’m just gonna make a proper vfs

#

And then the tmpfs

#

Then fat

#

I’m making a vfs call table now

#

vfs_call_table_t 🗣️🗣️

small hound
#

what's with all the _t naming for structs

bold grove
#

Not all my structs are named that way tho

#

Like my page info struct is just struct Page

small hound
#

idk your choice

bold grove
#

I do reserve _t for type definitions

small hound
#

so why are you also using it for struct definitions

bold grove
#

I’m not

#

I’m making my vfs call table a type

small hound
bold grove
#

Actually yeah I guess you’re right I’ll just call it vfs_call_table

#

So I’ll make a vfs_handle struct

#

Then a vfs mount point struct

#

I’ll actually try to document this process better

#

Bc I have been studying on vfs’s a lot

bold grove
#

Okay so I’ve made the structs maybe

radiant grove
#

:o

bold grove
#

I made a handle, file system, call table

radiant grove
#

congrats

bold grove
#

getting somewhere

small hound
#

why are create/delete on mountpoints

#

also copy and clone as vfs level operations is kinda silly

bold grove
#

Oh yeah you’re right, I was gonna create a entries in the table to destroy and create a file system

bold grove
small hound
small hound
#

code bloat

bold grove
small hound
#

you can make copy/clone with a create() read() and write()

small hound
#

userland utilities that perform such functionality do not rely on underlying system calls to do stuff like copy()

#

they just use other functions and create copy from it

small hound
bold grove
#

Yeah but what if I wanted to clone a device or file in the vfs

#

Not from userland

small hound
bold grove
#

Ah okay I see

small hound
#

so sure you can have a vfs_copy() that just relies on the other ones

bold grove
#

Oki I’ll get rid of them

#

Okay I see

bold grove
small hound
#

and the underlying filesystem doesn't need to provide those

bold grove
#

Im almost done creating the structs

bold grove
bold grove
bold grove
small hound
#

I am starting to have a wee bit of doubt that you actually "finish" these things when you post messages that you do

#

congrats if you actually did it tho lol

bold grove
#

wanna see?

small hound
bold grove
#

im not gonna commit it just yet

#

but ill put in paste bin

#

im not done yet

#

becausei have to make the init function but

#

i think its decently solid

bold grove
small hound
bold grove
#

im just missing one function

small hound
bold grove
#

i finished the op table

bold grove
#

did the rest of it look okay?

small hound
#

no, you are leaking memory

bold grove
#

where

small hound
#

nothing is refcounted

#

you just leak memory

bold grove
#

oh i forgot some free statement i know that

small hound
#

no you forgot a refcount

#

so you just leak memory

bold grove
#

should i just add a ref count to each vfs_handle

#

or what

small hound
#

it also just... isn't thread safe

bold grove
#

well yeah i know that much

#

i will fix that

#

so each time i reference a node anywhere

#

i should just increment the ref count?

#

should it just be like this ?

small hound
#

no it needs to be atomic

bold grove
#

just an atomic uint?

#

for each node?

small hound
#

with more steps to it

# 
static inline bool refcount_inc_not_zero(refcount_t *rc) {
    for (;;) {
        unsigned int old = atomic_load(rc);
        if (old == 0)
            return false;

        unsigned int expected = old;
        if (atomic_compare_exchange_weak(rc, &expected, old + 1))
            return true;

        old = expected;
    }
}

static inline bool refcount_dec_and_test(refcount_t *rc) {
    for (;;) {
        unsigned int old = atomic_load(rc);
        if (old == 0) {
            k_panic("%s(): Possible UAF!\n", __func__);
            return false;
        }

        unsigned int expected = old;
        if (atomic_compare_exchange_weak(rc, &expected, old - 1))
            return (old - 1) == 0;

        old = expected;
    }
}```
bold grove
#

when would you decrement a ref count

small hound
bold grove
#

okay i see

#

im making it thread safe now

small hound
#
  • you can have an explicit destroy function so if some process has a bunch of handles open it goes and yeets all of them at the end of the process's life
#

this way userspace can't leak memory by not close()ing stuff

bold grove
#

so my vfs struct should have a lock

small hound
#

wdym where would you use it

bold grove
#

no like lets say i have a tmpfs

#

and i acknowledge it

#

should each filesystem have a lock within the vfs

#

or should it just be a vfs lock

small hound
#

wdym by this

bold grove
#

like how would i prevent race

#

in the vfs operations

small hound
#

the presence of a lock doesn't mean anything unless the use of the lock is stated

#

what kinds of vfs operations do you mean

bold grove
#

open for example

small hound
#

if you're doing global modification to the vfs then sure, like modifying the mount table

bold grove
#

yeah

small hound
#

you do not want to acquire this global lock for opening files because that is really pointless

#

unless you are somehow doing some large modifications of the global vfs structures on open()

bold grove
#

but i should be locking for shared states

#

like the mount table

#

so for example

# 
static void vfs_add_mountpoint(struct vfs_mountpoint* mp) {
    if (mp_list.tail == NULL) {
        mp_list.tail = mp;
    } else {
        mp_list.tail->next_mountpoint = mp;
    }
    mp_list.tail = mp;
}

static void vfs_remove_mountpoint(struct vfs_mountpoint* mp) {
    struct vfs_mountpoint *m;
    if (mp == mp_list.tail) {
        mp_list.tail = mp->next_mountpoint;
    } else {
        for (m = mp_list.tail; m != NULL; m = m->next_mountpoint) {
            if (m->next_mountpoint == mp) {
                m->next_mountpoint = mp->next_mountpoint;
                break;
            }
        }
    }
}
#

i should probably lock

#

while modifying the mountpoint table

small hound
#

yes and you should probably not use a singly linked list here and should use a hashtable instead

bold grove
#

not singly linked list

small hound
#

why is there a separate fs list

bold grove
#

why wouldnt there be

small hound
#

because why do you need it?

#

what's it for

bold grove
#

for example

# 
struct vfs_fs *vfs_find_fs(int type) {
    struct vfs_fs *fs_to_find;
    for (fs_to_find = fs_list.head; fs_to_find != NULL; fs_to_find = fs_to_find->previous) {
        if (fs_to_find->filesystem_type == type) {
            break;
        }
    }
    return fs_to_find;
}
#

to find an fs of a certain type

small hound
#

you can also just use your mount table to do this

#

no need for a separate list

bold grove
#

i see

small hound
#

💥

bold grove
#

so within this funnction

# 
int vfs_acknowledge_fs(struct vfs_fs *fs) {


    if (!fs) {
        log("vfs_acknowledge_fs: fs is NULL\n", RED);
        return -1;
    }
    fs->previous = fs_list.head;
    fs_list.head = fs;

    log_uint("vfs: acknowledged filesystem type ", fs->filesystem_type);
    return 0;
}


int vfs_unacknowledge_fs(struct vfs_fs *fs) {
    return 0; 
}
#

i should just add the filesystem to the mounpoint table?

#

instead of a seperate fs list

small hound
bold grove
#

i thought it would be a good idea t have a global list of file systems known to the vfs

small hound
#

didn't you read papers on this, which ones said to do this

I'm curious now

small hound
bold grove
#

you talked about a while ago having a struct globally with some info and i kinda started doing that everywhere

#

so i just thought it might come in handy

#

but youre right

#

its prob not very useful

small hound
#

lol

bold grove
#

is this looking better for the thread safety

#

@small hound

# 
static struct vfs_mountpoint* vfs_create_mountpoint(char* device, char* mount_point, int type) {
    struct vfs_mountpoint* mp = vfs_mp_struc_alloc();
    if (!mp)
        return NULL;

    spinlock(&mp_list.lock);
    if (vfs_assign_mp_fs(mp, type) != 0) {
        kfree(mp, sizeof(struct vfs_mountpoint));
        spinlock_unlock(&mp_list.lock, true);
        return NULL;
    }

    if (vfs_mp_path_alloc(mp, mount_point) != 0) {
        kfree(mp, sizeof(struct vfs_mountpoint));
        spinlock_unlock(&mp_list.lock, true);
        return NULL;
    }

    if (vfs_mp_dev_alloc(mp, device) != 0) {
        kfree(mp->mount_point, flopstrlen(mp->mount_point) + 1);
        kfree(mp, sizeof(struct vfs_mountpoint));
        spinlock_unlock(&mp_list.lock, true);
        return NULL;
    }

    spinlock_unlock(&mp_list.lock, true);
    return mp;
}

static void vfs_add_mountpoint(struct vfs_mountpoint* mp) {

    spinlock(&mp_list.lock);
    if (mp_list.tail == NULL) {
        mp_list.tail = mp;
    } else {
        mp_list.tail->next_mountpoint = mp;
    }
    mp_list.tail = mp;
    spinlock_unlock(&mp_list.lock, true);
}
small hound
#

seems a bit odd to just assume that interrupts/preemption would be enabled at the time of the lock and just arbitrarily pass in a true

bold grove
#

my spinlock by default has that parameter

#

which i will fix at some point yeah

small hound
#

what does it do

bold grove
#

restores interrupts if you disabled them

#

and by default spinlock() does

#

sorry yeah

#

kinda stupid naming

small hound
#

it's whatever as long as you can guarantee that this is only called when interrupts are enabled before acquiring the lock

bold grove
#

yeah it checks

small hound
#

otherwise things might be a bit whimsical

bold grove
#

i have a macro

#

to check if ints are enabled

small hound
small hound
bold grove
#

shit yeah

#

i should spinlock within the functions

#

not within create_mountpoint

small hound
#

so would do you lock the list in them

bold grove
#

oh yeah... i dont access the mplist within any of that

#

so theres no need

#

but here i would need to

# 
static void vfs_remove_mountpoint(struct vfs_mountpoint* mp) {
    struct vfs_mountpoint *m;

    spinlock(&mp_list.lock);

    if (mp == mp_list.tail) {
        mp_list.tail = mp->next_mountpoint;
    } else {
        for (m = mp_list.tail; m != NULL; m = m->next_mountpoint) {
            if (m->next_mountpoint == mp) {
                m->next_mountpoint = mp->next_mountpoint;
                break;
            }
        }
    }
    if (mp_list.tail == NULL) {
        mp_list.head = NULL;
    }

    spinlock_unlock(&mp_list.lock, true);
}
#

and within create_mp and remove_mp i should increment the ref count on create and decrement on remove?

small hound
small hound
#

refcount on what

bold grove
#

the mountpoint

#

does that need a ref count

#

or just nodes

small hound
bold grove
#

is it fine to do a singly linked list for now, for the mountpoints

small hound
#

uhhhh I just suggested a hashtable but if you really wanna do that then sure lol

bold grove
#

no i know, but i just wanna get this working first then do the hashtable

small hound
#

hashtable just simpler

#

sure a list works

#

you can use a list

bold grove
small hound
#

ya it's going pretty good

bold grove
#

like i really tried to do hella research and

#

im also doing one other thing you suggested

small hound
#

yes 👍👍👍👍👍 you're doing great 😃😄😃😃

bold grove
#

that was the self documenting function

#

to split up big ahh functions

small hound
#

sory if I come off wrong or mean or anything lol, I'm tryna help so that this project goes smoothly further down the line

bold grove
#

you just questioned stuff you saw which is fair and what i need

small hound
#

oh ok 👍

bold grove
#

i ping you alot because you spot stuff i dont

small hound
#

👍 no problem 😁

bold grove
# 
typedef struct vfs_refcount {
    atomic_int refcount;
    
} vfs_refcount_t;
#

🗣️

small hound
#

real but you just wanna make a general purpose refcount here

bold grove
#

wdym?

#

not atomic?

small hound
#

instead of a custom vfs refcount struct

bold grove
#

ahhh i see

#

should i just make a fule in lib/

#

called refcount.h

#

so i can use it everywhere

small hound
#

sure yea

#

you can just nab the implementations I sent

#

you technically don't need the CAS loop, I just did it because you can catch a UAF with it better

#

I'm sure you already know how refcounts work but here's the refresher

  • you initialize a refcount, usually with a reference of one to represent the initial reference
  • as objects are passed around, refcounts fluctuate (inc/dec), and stuff happens
  • once an object's refcount hits zero, you free it
#

and in the CAS loop I just check if the refcount hits zero before we get to decrement it because that would indicate some double decrement on a refcount

bold grove
#

so i would init like this

small hound
#

indeed

#

you can also just have a standard refcount_inc that just becomes the atomic fetch add

#

if you want to you can guard against overflow but a refcount of 4 billion probably won't be reached

bold grove
#

i might as well have the one here

#

i wont fix it if its not broken

small hound
# 
static inline bool refcount_inc(refcount_t *rc) {
    for (;;) {
        unsigned int old = atomic_load(rc);
        if (old == UINT_MAX)
            return false;

        unsigned int expected = old;
        if (atomic_compare_exchange_weak(rc, &expected, old + 1))
            return true;
    }
}``` you can panic on refcount overflow here probably
small hound
# bold grove i might as well have the one here

the reason behind inc_notzero and dec_andtest is because those guard against the uaf of a refcount already being zero and not becoming zero from the caller/the caller not getting the chance to increment it

#

meaning someone else set it to zero and we didn't get the chance to decrement/increment our refcount so it would uaf

bold grove
#

porbably better to prevent against that case then

#

okay so heres this

#

i think thats all i need in the init function

#

oh shit

#

i set null twice

small hound
#

doesn't seem to be doing anything lol

#

and also you wouldn't refcount "the list as a whole"

bold grove
#

yeah youre right lol

small hound
#

you'd do it per object with a lifetime

#

since the list has a static lifetime

bold grove
#

so for each new object i would need a refcount_t

small hound
#

for each object that needs it, yea

bold grove
small hound
#

sounds great

bold grove
#

ill push the code to github

small hound
#

sounds great 👍

bold grove
#

its very rough but it works

bold grove
#

lmk what you think

small hound
#

ok I'll probably check later, it late rn

bold grove
#

okay have a good night

bold grove
#

Okay time to implement FAT

bold grove
#

I’m so hypeddddddddd

#

Imma fix vfs

#

Clean it up

torpid viper
#

or else you end up with a lot of things to clean

#

and it take month

small hound
#

their codebase is all over the place

#

💥

bold grove
radiant grove
#

linux probably had more programmers over its lifespan than windows ever did

#

its a shitfest

small hound
small hound
#

🤪

bold grove
#

Okay imma start making the fat implementation

bold grove
#

okay i stepped back and im doing tmpfs first

bold grove
#

okay

#

herre we go

#

We registered the tmpfs with the vfs

#

And I decided to only expose the tmpfs through the vfs op table

bold grove
#

i added ref counting

#

and i made mount and unmount thread safe

#

still got more but

#

and yes i know the way i initialize my spinlocks is weird as fuck

radiant grove
#

oh no theoretical conversions of file systems into a standardized format for easier access

bold grove
radiant grove
#

i figured

bold grove
#

okay all the main opreations are thread safe

#

I pushed the tmpfs

radiant grove
#

yay

#

i need to get paging done on my os now

bold grove
bold grove
bold grove
radiant grove
bold grove
#

Wait why did I link the header

#

There

radiant grove
#

im gonna die if i dont have paging my total sucking is beyond major

radiant grove
bold grove
#

It’s mainly just synchronization and node shit

radiant grove
#

fair

bold grove
#

You will probably learn more reading the vfs

#

Damnnnn commit schedule kinda fire recently

radiant grove
#

holy

#

indeed

small hound
# bold grove

formatting commits can make grepping through them easier

image.png
#

just suggestion 👍

bold grove
paper cloak
#

I use conventional commits

bold grove
paper cloak
bold grove
#

I should probably do those

bold grove
#

@thorn dust would you be able to read through my vfs and tmpfs and lmk what needs fixing and what could be done better

#

Here they are

small hound
#

(node->vfs_mode & VFS_MODE_APPEND) == VFS_MODE_APPEND seems weird

#

no need for the comparison operator

#

also the locks aren't right

bold grove
#

How are they not

small hound
#

you only lock for metadata modification not for file data operations unless explicity specified

#

and that data locking occurs at the page cache level

bold grove
#

For now file data is just an array of page pointers

#

Wait which one are you looking at

#

Vfs?

small hound
#

yea but you're again not supposed to lock the node when you read/write from and to its data

#

tmpflopfs.c

bold grove
#

Okay

small hound
#

also you'd probably wanna switch to a mutex instead of a spinlock here

bold grove
#

I haven’t implemented those yet but I will

#

But yeah i know a spinlock is not ideal

#

Spinning on a file operation is cancer

bold grove
small hound
#

metadata modification

#

size, gid, uid, mode, etc.

bold grove
#

And that’s it

paper cloak
small hound
#

and any other metadata modification

paper cloak
#

bro is trolling

bold grove
#

Yes I know the relative include paths are cancer but I prefer them

small hound
#

me when my code completely stops compiling after I move source files around

#

also uhhhh

static int tmpfs_unmount(struct vfs_mountpoint* mp, char* device) {
    (void)device;
    tmpfs_super_t* sb = (tmpfs_super_t*)mp->data_pointer;
    if (!sb) return -1;
    bool interrupt_state = spinlock(&sb->lock);
    if (refcount_dec_and_test(&sb->refcount)) {
        tmpfs_free_tree(sb->root);
        kfree(sb, sizeof(*sb));
        mp->data_pointer = NULL;
    }
    spinlock_unlock(&sb->lock, interrupt_state);
    return 0;
}```
#

me when i leak memory 🤪

#

if dec_and_test is false and it isn't freed you just leak memory because other dec_and_tests dont free the filesystem

#

ideally here you just drop the last reference and that automatically frees it

#

you'd want get() and put() helpers to do this

# 
static inline bool thread_get(struct thread *t) {
    return refcount_inc_not_zero(&t->refcount);
}

static inline void thread_put(struct thread *t) {
    if (refcount_dec_and_test(&t->refcount)) {
        if (atomic_load(&t->state) != THREAD_STATE_TERMINATED) {
            k_panic("final ref dropped while thread not terminated\n");
        }
        thread_free(t);
    }
}``` example
bold grove
#

Ahhh okay I see

#

There’s also undefined behavior in there because I free sb then unlock it

small hound
#

you can take it one step further here since you have locks and make acquire and release that get and put respectively, as well as acquire/release the lock

# 
static inline bool thread_acquire(struct thread *t) {
    if (!refcount_inc_not_zero(&t->refcount))
        k_panic("UAF");
    return spin_lock(&t->lock);
}

static inline void thread_release(struct thread *t, bool iflag) {
    spin_unlock(&t->lock, iflag);
    thread_put(t);
}``` example
paper cloak
bold grove
#

So i should make a tmpfs_lock_acquire or something

paper cloak
#

I much prefer retain/release

small hound
#

i like how they sound

#

very 'pokey' idk

small hound
bold grove
#

I would prob prefer retain and release

small hound
#

okay sure

#

same thing

bold grove
#

But yeah

#

I can do that

bold grove
#

And for the vfs

small hound
#

uh ima take a gander rq

#

op table looks very very incomplete

#

like 70% incomplete

bold grove
#

I know it doesn’t have everything yeah

small hound
#

why do your operations take in paths?

# 
struct vfs_op_tbl {
    struct vfs_node* (*open) (struct vfs_node*, char*); // node, path
    int              (*close)(struct vfs_node*);        // node

    rw  read;  // node, buffer, size
    rw  write; // node, buffer, size
    
    void*  (*mount)  (char*, char*, int);
    int    (*unmount)(struct vfs_mountpoint*, char*); // mp, device name
    int    (*create) (struct vfs_mountpoint*, char*); // mp, file name
    int    (*delete) (struct vfs_mountpoint*, char*); // mp, file name
    int    (*rename) (struct vfs_mountpoint*, char*, char*); // mp, old name, new name
    int    (*ctrl)   (struct vfs_node*, unsigned long, unsigned long); // node, command, arg
    int    (*seek)   (struct vfs_node*, unsigned long, unsigned char); // node, offset, whence

    struct vfs_directory_list* (*listdir)(struct vfs_mountpoint*, char*); // mp, path


};```
#

why is the filesystem responsible for parsing a path?

#

create/delete/etc should take in a node not a mountpoint here

paper cloak
#

you should have a lookup function that works on path segments

small hound
#

yus

bold grove
bold grove
paper cloak
#

why

#

why

small hound
#

to be silly with a colon three

bold grove
#

I was planning on adding security features within tmp alloc and free but

#

Never ended up doing that

bold grove
paper cloak
#

like what

bold grove
paper cloak
#

yes

bold grove
#

Okay note taken

bold grove
bold grove
paper cloak
#

lol

small hound
bold grove
#

I didn’t even see that shit

paper cloak
#

omfg

small hound
#

"What's a signed integer"

bold grove
#

Dude he opened an issue but didn’t even star the repo

bold grove
small hound
#

the interviewer said call stacks are an OS level concept

#

crazy

paper cloak
#

no this guy is a moron

#

he asks ambiguous questions to feel superior

bold grove
small hound
#

yes you heard that right and he also said they're vastly different from the stack data structure

bold grove
bold grove
#

We need the plate analogy

#

The one Wikipedia uses lol

small hound
bold grove
paper cloak
#

coding jesus is his channel

small hound
paper cloak
bold grove
paper cloak
#

yes

bold grove
#

What the fuck

#

I don’t even have to watch the video he just put all the bullshit for me

paper cloak
#

the answer to most of those is "it depends"

bold grove
#

Deadass bro

small hound
bold grove
#

What is cache? Idk what the fuck type of cache do you mean

small hound
bold grove
#

Like we have cpu caches, page caches, browser caches, etc

#

How many layers

paper cloak
#

BRO IT DEPENDS

bold grove
#

There is literally

#

In c

#

Integers that tell you

paper cloak
#

is the system ILP64 or LP64?

small hound
bold grove
#

How big they are

small hound
#

and then they are 4 bytes literally everywhere and everyone forgets that they are only defined as being at least 2 bytes wide

bold grove
#

why are these questions relevant

They aren’t

bold grove
#

C is archaic as fuck though so

#

That makes sense

small hound
bold grove
small hound
bold grove
#

I know bro

#

Deadass

#

Listening to this makes me feel smart

#

I’m running on the treadmill rn listening to it on my tv

#

Max dopamine

small hound
#

I'm about to go running stay fit queens

bold grove
#

Yessir have fun

thorn dust
small hound
#

he says it's because he "used to look like jesus"

thorn dust
small hound
#

meanwhile the guy just spreads misinformation online lol

#

it's kind of impressive to consistently be so confidently incorrect

bold grove
bold grove
small hound
#

like asking what 64 bit means and saying that it's the register bitwidth without specifying that it's the general purpose register bitwidth

#

x86 (with avx512) is my favorite 512 bit architecture 🗣️

bold grove
#

Wait till bro finds out not all registers are 64 bit

#

I doubt he would even know the acronym gpr

bold grove
#

So bad

small hound
#

the humble tlb shootdown:

bold grove
#

Even if you said it in plain English

#

Symmetric multiprocessing

#

He wouldn’t know

viscid zephyr
radiant grove
bold grove
#

Dude there was a fucking shooting where I picked up my friend in Spokane

torpid viper
radiant grove
#

u good?

radiant grove
radiant grove
bold grove
small hound
bold grove
bold grove
bold grove
#

Imma take a little break guys

#

Not feeling well

bold grove
#

Like I was tryna get into my friends place before we went out and as he was unlocking the door there was a shot and then I heard the bullet bounce off the wall nearby me

#

And we saw the guy leave the scene

#

I feel like I’m actually lucky

#

To be here

#

Because I literally heard the shooters breathing a few meters from me

#

So I’m kinda not in a good state rn lol

radiant grove
#

oh damn

#

i get you its all good

bold grove
#

yall are gonna love this

bold grove
#

I’m gonna be gone for a couple weeks. I’m going into impatient treatment

#

Last night I had an extremely bad acid trip and freaked out and had a psychotic episode

small hound
#

🙏🙏🙏🙏 get well soon

bold grove
#

GUYSSSSSSSSSSSSSS

#

IM BACKKKK

bold grove
#

Okay well I’m working now

#

I’m osdeving so fucking hard right now

#

I rewrote so much shit

#

Yall don’t even know

#

I’m rewriting so much shit

radiant grove
bold grove
radiant grove
bold grove
#

Yes

bold grove
#

Haven’t pushed it yet but

radiant grove
#

truly gdt of all time

#

my paging is still farts

bold grove
#

Why

radiant grove
#

cant work on it yet tho

bold grove
#

Show me

radiant grove
bold grove
#

Oh I see

radiant grove
#

i know how to make it work but unfinished

#

my kernel is somehow 27 mb

#

appauling

bold grove
#

How

#

Like your iso

radiant grove
#

idk????

bold grove
#

Or your kernel binary

#

Well what is 27mb

#

What file

#

Kernel

#

Or iso

#

Because the iso will be big

radiant grove
#

it says the linker end symbol is at 27 mb

#

i need a few pages

#

quite a few

bold grove
#

Hm

#

Strange

radiant grove
#

idk

#

big stack?

bold grove
#

It’s prob just multiboot

#

How big is your stack

radiant grove
#

idk

bold grove
#

What

#

How do you not know

#

How big your kernel stack is

radiant grove
#

im not home i cant check

#

i remember checking but i forgot the number

#

its big as shit tho iirc

#

takes most of the 27 mb

bold grove
#

Okay so

#

In your entry assembly file

#

Do you reserve a stack

radiant grove
#

yis

#

i think so

#

lemme check the githuv

#

github

bold grove
#

This is what I do

#

I align and do 8kb

#

It should be in your bss section

radiant grove
bold grove
#

BRO

radiant grove
#

i do uh

#

damn

bold grove
#

DAWG

#

why is your stack so huge

radiant grove
#

wtf was i smoking

#

this is at least half the universe

radiant grove
#

idk

#

deadass wtf was i smoking

#

my old code is fucking SHIT

radiant grove
#

im gonna do 8192

bold grove
#

Yes

#

It is more than enough

radiant grove
#

27 fucking mb stack

#

damm

#

insane

#

maybe thats why paging shits itself

#

wtf is that stack

#

anyway are u gonna rewrite idt next?

bold grove
#

No

#

I’m fine with mine

#

I’m rewriting some other stuff tho

radiant grove
#

what do u want to rewrite?

bold grove
#

Mmmm

#

Vfs

#

Even tho I just wrote it

radiant grove
#

big unspaghetti

bold grove
#

It’s because a lot of my operations take mountpoinrs

#

And walk to paths

#

Rather than taking in nodes

#

So

radiant grove
#

oh

bold grove
#

That’s kinda fucked

radiant grove
#

truly

#

anything that has "virtual" in it code for it ends up looking like total farts

#

or is it just my experience with shitty tutorials and 27 mb stacks

paper cloak
radiant grove
#

inline asm

small hound
radiant grove
small hound
#

u just dont understand

unborn latch
#

thats 128kib

bold grove