OBOS (not vibecoded) | Operating System Development | Page 18

flint idol Sep 2, 2024, 1:03 AM

#

the thread would be indefinitely blocked waiting for the event to be set

#

was quite fun to track that one down...

#

https://tenor.com/view/the-return-of-the-king-lord-of-the-rings-ro-tk-title-card-gif-22191928

Tenor

flint idol Sep 2, 2024, 1:05 AM

#

flint idol the thread would be indefinitely blocked waiting for the event to be set

the solution was just to lower the irql after the thread is unblocked

#

obos_status OBOS_Kill(struct thread* as, struct thread* thr, int sigval);
obos_status OBOS_SigAction(int signum, const sigaction* act, sigaction* oldact);
obos_status OBOS_SigSuspend(sigset_t mask);
obos_status OBOS_SigPending(sigset_t* mask);
obos_status OBOS_SigProcMask(int how, const sigset_t* mask, sigset_t* oldset);
obos_status OBOS_SigAltStack(const uintptr_t* sp, uintptr_t* oldsp);

void OBOS_SyncPendingSignals(size_t maxCount);

void OBOSS_RunSignal(int sigval, sigaction* what);```

#

I have all my signal api done

#

bottom two functions are internal

#

2nd-last simply runs all signals in the current thread

#

last should be trivial to understand

#

although the last function will probably change a bit

#

the last function will take in a siginfo_t

#

time to start with the most trivial function to implement

#

kill

#

thr->signals.pending |= BIT(sigval);

#

bam

#

done

#

well

#

there is also setting some metadata

#

obos_status OBOS_Kill(struct thread* as, struct thread* thr, int sigval)
{
    if (!as || !thr || !(sigval >= 0 && sigval <= SIGRTMAX))
        return OBOS_STATUS_INVALID_ARGUMENT;
    if (thr->signal_info->pending & BIT(sigval))
        return OBOS_STATUS_SUCCESS;
    thr->signal_info->pending |= BIT(sigval);
    thr->signal_info->signals[sigval].addr = nullptr;
    thr->signal_info->signals[sigval].status = 0;
    thr->signal_info->signals[sigval].udata = 0;
    thr->signal_info->signals[sigval].sigcode = 0;
    thr->signal_info->signals[sigval].sender = as;
    return OBOS_STATUS_SUCCESS;
}```

#

easy

#

the fields set to zero should be setup if needed after Kill is called

#

obos_status OBOS_SigSuspend(sigset_t mask)
{
    Core_MutexAcquire(&Core_GetCurrentThread()->signal_info->lock);
    sigset_t old = Core_GetCurrentThread()->signal_info->mask;
    Core_GetCurrentThread()->signal_info->mask = mask;
    Core_WaitOnObjectsPtr(
        64, 
        sizeof(Core_GetCurrentThread()->signal_info->signals[0]), 
        (struct waitable_header*)&Core_GetCurrentThread()->signal_info->signals
    );
    Core_GetCurrentThread()->signal_info->mask = old;
    Core_MutexRelease(&Core_GetCurrentThread()->signal_info->lock);
    return OBOS_STATUS_SUCCESS;
}```

#

hopefully this is signal-safe

#

well since signal_info->lock is always taken when manipulating signals

#

I should be fine

#

wait...

#

I can't use wait on objects ptr

#

I'll just change this up a bit

#

this is better:

obos_status OBOS_SigSuspend(sigset_t mask)
{
    Core_MutexAcquire(&Core_GetCurrentThread()->signal_info->lock);
    sigset_t old = Core_GetCurrentThread()->signal_info->mask;
    Core_GetCurrentThread()->signal_info->mask = mask;
    Core_WaitOnObject(WAITABLE_OBJECT(Core_GetCurrentThread()->signal_info->event));
    Core_GetCurrentThread()->signal_info->mask = old;
    Core_MutexRelease(&Core_GetCurrentThread()->signal_info->lock);
    return OBOS_STATUS_SUCCESS;
}```

rapid pulsar Sep 2, 2024, 1:46 AM

#

Don't forget that the libc version should always fail :-)

flint idol Sep 2, 2024, 1:47 AM

#

yeah so I decided to just implement it in the kernel

rapid pulsar Sep 2, 2024, 1:47 AM

#

No I mean the function is defined to always fail with EINTR

flint idol Sep 2, 2024, 1:47 AM

#

ah

#

yeah I forgor

#

yeah I'll do that when I port mlibc

rapid pulsar Sep 2, 2024, 1:48 AM

#

Unless you gave it a bad sigset, then it can fail with a fault, but the "success" case is that it gives EINTR because that's literally what the caller is asking it to do (wait until it gets interrupted).

flint idol Sep 2, 2024, 1:49 AM

#

yup

#

your help is much appreciated, ty

rapid pulsar Sep 2, 2024, 1:50 AM

#

Just gotta follow up myself ;) even my current signal implementation was written largely to 'practice what i preach' helping someone else with it a while back

flint idol Sep 2, 2024, 2:03 AM

#

obos_status OBOS_Kill(struct thread* as, struct thread* thr, int sigval);
obos_status OBOS_SigAction(int signum, const sigaction* act, sigaction* oldact);
obos_status OBOS_SigSuspend(sigset_t mask);
obos_status OBOS_SigPending(sigset_t* mask);
enum {
    SIG_BLOCK,
    SIG_SETMASK,
    SIG_UNBLOCK,
};
obos_status OBOS_SigProcMask(int how, const sigset_t* mask, sigset_t* oldset);
obos_status OBOS_SigAltStack(const uintptr_t* sp, uintptr_t* oldsp);```

#

these are all implemented

#

all that's left is:

void OBOS_SyncPendingSignals(size_t maxCount);

void OBOSS_RunSignal(int sigval, sigaction* what);```

#

with run signal being arch-specific

inland radish Sep 2, 2024, 10:01 AM

#

why not?

rapid pulsar Sep 2, 2024, 10:07 AM

#

inland radish why not?

The macros shall expand to positive integer constant expressions with type int and distinct values. The value 0 is reserved for use as the null signal

For one, because POSIX says so. But also this just holds in general and you’re bound to run into issues.

flint idol Sep 2, 2024, 10:12 AM

#

flint idol all that's left is: ```c void OBOS_SyncPendingSignals(size_t maxCount); void OB...

I have figured out that OBOS_SyncPendingSignals can be simplified to just call one signal

#

and return

#

and run signal should take in an interrupt frame+thread+sigval and output a context

#

and after signals I will be optimizing swap

#

well more like my vmm

#

but anyway

inland radish Sep 2, 2024, 10:19 AM

#

rapid pulsar > The macros shall expand to positive integer constant expressions with type int...

what's the null signal

flint idol Sep 2, 2024, 10:20 AM

#

probably means "nothing"

flint idol Sep 2, 2024, 10:33 AM

#

flint idol and run signal should take in an interrupt frame+thread+sigval and output a cont...

idk why I said output a context

#

anyway I think ucontext_t can just be typedef'd to thread_context_info

flint idol Sep 2, 2024, 10:44 AM

#

flint idol all that's left is: ```c void OBOS_SyncPendingSignals(size_t maxCount); void OB...

for those I'll need to implement some memcpy functions for kernel->usermode and usermode->kernel (I plan on implementing KPTI, so I can't directly do a memcpy)

#

so if it's user to kernel, I can do CoW on the memory regions after the first page

#

up to the second last or last page depending on if the last byte is page aligned or not

#

for kernel->user I could do the same, but for security reasons I won't

#

(don't want to accidentally leak kernel memory to user mode)

flint idol Sep 2, 2024, 11:29 AM

#

flint idol so if it's user to kernel, I can do CoW on the memory regions after the first pa...

maybe that's not worth the effort

flint idol Sep 2, 2024, 12:04 PM

#

my kasan is so slow and so memory inefficient

#

that when I remove it

#

the kernel uses 7M less memory

#

and boots a LOT faster

flint idol Sep 2, 2024, 12:05 PM

#

flint idol the kernel uses 7M less memory

*4M

#

or maybe it's a lot faster because a symbol is undefined in a driver

#

causing it to not do anything

#

turns out the ahci driver was using ~3M

#

but it's still a lot faster without kasan

#

but anyway I should be able to copy to and from user pages

flint idol Sep 2, 2024, 12:27 PM

#

dam the log is so much more cluttered now

#

which is a good thing

#

because that means I didn't have ~25K changes in the summer for nothing

#

23K in august

#

and 5K in july

#

anyway, now that all public api for signals is done

#

I will start on the implementation

#

void OBOSS_SigReturn(interrupt_frame* frame);
void OBOSS_RunSignal(int sigval, thread* on, interrupt_frame* frame);```

#

I haven't written arch-specific code for quite a bit

#

almost just put a privellage escalation bug in my signal code

#

silly me!

#

basically I was doing:

memcpy(frame, &ctx.frame, sizeof(*frame));

with ctx.frame coming from usermode

#

so a user mode program could abuse that to set cs&ss to kernel code+data segments

#

which is why we must enable SMAP+SMEP

#

but then they could also abuse rip and set it to a kernel symbol address

#

and if I had some sort of internal set process uid function

#

they could use that to escalate a process to root

#

void OBOSS_SigReturn(interrupt_frame* frame)
{
    // Use frame->rsp to restore the previous thread context.
    thread_ctx ctx = {};
    memcpy_usr_to_k(&ctx, (void*)frame->rsp, sizeof(ctx));
    if (ctx.frame.rip >= OBOS_KERNEL_ADDRESS_SPACE_BASE || frame->rsp >= OBOS_KERNEL_ADDRESS_SPACE_BASE)
        return;
    wrmsr(KERNEL_GS_BASE, ctx.gs_base);
    wrmsr(FS_BASE, ctx.fs_base);
    memcpy(frame, &ctx.frame, sizeof(*frame));
    frame->cs = 0x18|3;
    frame->ss = 0x20|3;
    frame->ds = 0x20|3;
    frame->rflags |= RFLAGS_INTERRUPT_ENABLE;
    frame->rflags |= RFLAGS_IOPL_3;
    frame->cr3 = CoreS_GetCPULocalPtr()->currentContext->pt;
}```

#

hopefully no one tries to exploit this...

#

worst this can cause is the process getting SIGSEGV because the return address is invalid (hopefully)

real pecan Sep 2, 2024, 12:59 PM

#

flint idol hopefully no one tries to exploit this...

I will 🙂

flint idol Sep 2, 2024, 12:59 PM

#

you'll report the bug, right

real pecan Sep 2, 2024, 12:59 PM

#

yes Kapp

flint idol Sep 2, 2024, 1:03 PM

#

I'm sure it's stable enough

weary hound Sep 2, 2024, 1:12 PM

#

flint idol ```c void OBOSS_SigReturn(interrupt_frame* frame) { // Use frame->rsp to res...

this check ```c
if (ctx.frame.rip >= OBOS_KERNEL_ADDRESS_SPACE_BASE || frame->rsp >= OBOS_KERNEL_ADDRESS_SPACE_BASE)
return;

#

also you want to clear iopl not set it?

flint idol Sep 2, 2024, 1:13 PM

#

shi I did

flint idol Sep 2, 2024, 1:13 PM

#

weary hound this check ```c if (ctx.frame.rip >= OBOS_KERNEL_ADDRESS_SPACE_BASE || frame...

oh ye

#

I coulda sworn I had a good reason to put that there

weary hound Sep 2, 2024, 1:13 PM

#

also also you check it after you try to copy

flint idol Sep 2, 2024, 1:14 PM

#

void OBOSS_SigReturn(interrupt_frame* frame)
{
    // Use frame->rsp to restore the previous thread context.
    thread_ctx ctx = {};
    if (obos_is_error(memcpy_usr_to_k(&ctx, (void*)frame->rsp, sizeof(ctx))))
        return;
    Core_EventClear(&Core_GetCurrentThread()->signal_info->event);
    wrmsr(KERNEL_GS_BASE, ctx.gs_base);
    wrmsr(FS_BASE, ctx.fs_base);
    memcpy(frame, &ctx.frame, sizeof(*frame));
    frame->cs = 0x18|3;
    frame->ss = 0x20|3;
    frame->ds = 0x20|3;
    frame->rflags |= RFLAGS_INTERRUPT_ENABLE;
    frame->cr3 = CoreS_GetCPULocalPtr()->currentContext->pt;
}```

#

this is what it is now btw

weary hound Sep 2, 2024, 1:14 PM

#

this still lets the user set iopl=3 on their own?

flint idol Sep 2, 2024, 1:14 PM

#

oh yeah

#

I just need to add the xrstor to the function

#

or fxrstor

#

depends

#

but anyway

weary hound Sep 2, 2024, 1:18 PM

#

weary hound this still lets the user set iopl=3 on their own?

managarm does ```cpp
// Allow modifying the normal non-privileged flags.
constexpr uintptr_t allowedFlagsMask = 0b1000011000110111111111;
thread->_executor.general()->rflags &= ~allowedFlagsMask;
thread->_executor.general()->rflags |= regs[17] & allowedFlagsMask;

flint idol Sep 2, 2024, 1:19 PM

#

I see

#

you guys won't mind if I just... take that, right

flint idol Sep 2, 2024, 1:46 PM

#

I'll implement m68k signals later™️

#

but I should have the basis for signals done

vernal chasm Sep 2, 2024, 2:10 PM

#

rapid pulsar > The macros shall expand to positive integer constant expressions with type int...

imagine actually following posix O_O

flint idol Sep 2, 2024, 2:11 PM

#

only reason my signals are going to be posix compliant is because I don't want to spend a lot of time on designing them

#

just for them to be shit anyway

#

and hard to port in mlibc

#

and no I'm not going to make my own libc

vernal chasm Sep 2, 2024, 2:12 PM

#

flint idol and no I'm not going to make my own libc

sadcat

#

Why not, it's such a simple thing to do

flint idol Sep 2, 2024, 2:13 PM

#

SIGILL is illegal instruction right

#

and do I use SIGSEGV for #GP

vale nymph Sep 2, 2024, 2:15 PM

#

Yes for both

flint idol Sep 2, 2024, 2:15 PM

#

ok

inland radish Sep 2, 2024, 2:26 PM

#

flint idol which is why we must enable SMAP+SMEP

eh, you don't really NEED to do that

inland radish Sep 2, 2024, 2:26 PM

#

flint idol ```c void OBOSS_SigReturn(interrupt_frame* frame) { // Use frame->rsp to res...

also why are you OR-ing 3 on the segment values

#

thats not how you set CPL, what does it do

flint idol Sep 2, 2024, 2:26 PM

#

I am 50% sure I'm supposed to

#

to set CPL

#

*RPL

inland radish Sep 2, 2024, 2:27 PM

#

what's RPL?

flint idol Sep 2, 2024, 2:27 PM

#

#

I think it's just the ring level of the selector

inland radish Sep 2, 2024, 2:28 PM

#

i see

#

so the fact that a segment selector is oddly similar to the offset of the specific segment in the GDT is only coincidence

flint idol Sep 2, 2024, 2:28 PM

#

indeed

#

out of curiosity, has boron gotten to ring 3 yet, or have you entirely been doing kernel mode dev

#

#

that's from the sdm

vernal chasm Sep 2, 2024, 2:36 PM

#

flint idol SIGILL is illegal instruction right

Nono the program has become sick

#

sad

inland radish Sep 2, 2024, 2:44 PM

#

flint idol out of curiosity, has boron gotten to ring 3 yet, or have you entirely been doin...

nope

#

i wanna get a semi decent Mm and Ps first

flint idol Sep 2, 2024, 2:44 PM

#

makes sense how you'd make that mistake

inland radish Sep 2, 2024, 2:44 PM

#

(memory manager, process system)

flint idol Sep 2, 2024, 2:44 PM

#

yup

#

gl

inland radish Sep 2, 2024, 2:44 PM

#

flint idol makes sense how you'd make that mistake

true

flint idol Sep 2, 2024, 2:44 PM

#

I like boron's codebase style

#

quite clean

#

so I've made all cpu exception handlers signal the thread

#

it happened on

#

so I guess that's posix signals done

#

except for SIGKILL

#

since I have no way to terminate processes

#

or even threads

#

I can terminate the current thread

#

but not remote threads

#

yet

inland radish Sep 2, 2024, 2:54 PM

#

flint idol I like boron's codebase style

thanks :)

flint idol Sep 2, 2024, 4:43 PM

#

marker

#

dam he explained it twice

#

so yeah to optimize my vmm I'll just do that

#

how hard could it possibly be

#

so yeah I basically have a dirty list

#

I throw dirty pages that are trimmed from the working set on there

#

or if no more references

#

then eventually

#

the kernel writes back the pages to the swap

#

and frees the physical pages

flint idol Sep 2, 2024, 5:27 PM

#

if I lock in I can start ttys today

#

but I first need to do this thing with the vmm

flint idol Sep 2, 2024, 8:24 PM

#

flint idol if I lock in I can start ttys today

with how it looks, I can probably start

#

researching them today

flint idol Sep 2, 2024, 9:03 PM

#

it almost™️ works

#

Assertion failed in function Mm_RunPRA. File: /home/oberrow/Code/obos/src/oboskrnl/mm/handler.c, line 374. !ctx->referenced.nNodes```

#

except for that

#

idk what's up with that

#

ctx->referenced.nNodes = -2

#

thonk

#

if (curr->workingSets > 0)
            {
                REMOVE_PAGE_NODE(ctx->workingSet.pages, &curr->ln_node);
                ctx->workingSet.size -= curr->prot.huge_page ? OBOS_HUGE_PAGE_SIZE : OBOS_PAGE_SIZE;
            }
            else 
                REMOVE_PAGE_NODE(ctx->referenced, &curr->ln_node); // it's in the referenced list```

#

that was it

#

omg this is so faster

#

rather I'm hallucinating

#

or this is genuinely faster

#

well that's probably because I don't actually ever wake the dirty page writer

#

and now it recursively locks

#

fixed that

#

to get the occasional deadlock

#

obos moment

#

fixed

#

I think

#

other than the fact that the ahci driver is suddenly broken

#

well it's more like the partition thingy

#

but it's more likely to be the ahci driver

#

fixed

#

it was Mm_VirtualMemoryProtect

#

fuck

#

this is 100 ms slower

#

50ms to 100 ms slower

flint idol Sep 2, 2024, 10:24 PM

#

ASAN Violation at ffffffff8005e44f while trying to read 1 bytes from 0xffff80003a1b84f9 (Hint: Use of memory block after free).```

#

nooo

#

wtf

#

how was this bug never caught before

#

if (curr->allocated)
    Mm_Allocator->Free(Mm_Allocator, curr, sizeof(*curr));
offset = curr->prot.huge_page ? OBOS_HUGE_PAGE_SIZE : OBOS_PAGE_SIZE;```

flint idol Sep 2, 2024, 10:42 PM

#

the difference between the new swap impl. and the old one is .032 ms, with the old one being faster

#

but the difference is probably a lot bigger if paging to disk instead of ram

#

it was a lot faster using the other swap thing

#

hmm it deadlocks later on though

#

not an artifact of the disk swap

#

nvm

#

it's probably an artifiact of switching to a new swap

#

but the problem is it doesn't always deadlock

#

even more weirdly, the page writer thread has the lock acquired

#

but it's blocked

#

but it never blocks itself before releasing the spinlock

#

I really gotta stop forgetting to init my event objects

#

so it was both that and a bug with event pulse

#

(or so it seems)

#

it was that

#

tomorrow after school I will probably start implementation of ttys

#

I now need to understand them

flint idol Sep 2, 2024, 11:42 PM

#

flint idol I now need to understand them

I might be able to start today (within the next 3-4 hours) if they're simple enough.

flint idol Sep 3, 2024, 11:08 PM

#

so I'm rewriting the vmm now

#

https://github.com/OBOS-dev/obos/tree/vmm-rewrite

GitHub

GitHub - OBOS-dev/obos at vmm-rewrite

I was bored... Contribute to OBOS-dev/obos development by creating an account on GitHub.

#

some parts won't be

#

expect this to take weeks because school

#

so how will I get this shit to not be shit

#

well it's not shit

#

it's just this that's causing trouble

#

https://github.com/OBOS-dev/obos/blob/master/src/oboskrnl/mm/page.h#L18

GitHub

obos/src/oboskrnl/mm/page.h at master · OBOS-dev/obos

I was bored... Contribute to OBOS-dev/obos development by creating an account on GitHub.

#

but the problem is

#

that's used everywhere in the vmm

#

and even in some places outside the vmm

real pecan Sep 3, 2024, 11:19 PM

#

hope u actually rewrite the vmm instead of the entirety of the kernel

flint idol Sep 3, 2024, 11:20 PM

#

I want to refrain from the latter

#

yeah no it's really only ever used in the vmm

#

rarely outside

#

I'll keep the external vmm api

#

Mm_VirtualMemoryAlloc/Free/Protect

#

and just rewrite it

#

but now I'm kinda confused though

#

because I was told by fadanoid (I think) to use a rb tree for vmm page keeping

#

or smth like that

#

but idk how to do that with page regions

#

actually nvm I'm just not braining correctly

#

#

why couldn't anyone

#

have warned me

#

back then

#

my brain isn'tbraining pcorfeswfesw

#

scorecly

#

correctly

#

which is why I'll look at keyronex

#

I just need to summon fadanoid first

#

"m68k"

#

typo

#

https://github.com/Keyronex/Keyronex/blob/master/kernel/vm/vmp.h#L45

#

https://github.com/Keyronex/Keyronex/blob/master/kernel/vm/vmp.h#L47

#

is this for one page

#

or for a range

#

I think the former?

#

nvm

#

it's the latter

#

uhh so change of plans

#

~~obos rewrite 6~~

#

not rly

#

uhh

#

fuck

#

forgor

#

ah yes

#

shit

#

I'm typing faster than I can think atm

real pecan Sep 3, 2024, 11:35 PM

#

noooooo

flint idol Sep 3, 2024, 11:35 PM

#

flint idol ~~obos rewrite 6~~

not this

#

I am NOT doing that again

flint idol Sep 3, 2024, 11:36 PM

#

flint idol shit

instead of rewriting the VMM

#

I am going to leave a TODO for the next person to come across this

#

and put a readme in src/oboskrnl/mm

#

saying

Abandon all hope, ye who enter here

#

I should optimize the struct's memory usage at the least

#

uintptr_t swapId : PTR_BITS;            // The page's swap allocation id. Only valid if pagedOut == true.
    size_t swap_off : PTR_BITS;             // The offset in the swap buffer. Only valid if pagedOut == true.```

#

that's a full 4ewrwrwrw

#

34r234

#

16 bytes

#

which I might be able to cut down to 8

#

if I slightly change swap

#

there's a 24-byte struct in there

#

which can be changed

#

probably

#

so I have no idea how to fix this

#

https://github.com/OBOS-dev/obos/blob/master/src/oboskrnl/mm/page.h#L18

GitHub

obos/src/oboskrnl/mm/page.h at master · OBOS-dev/obos

I was bored... Contribute to OBOS-dev/obos development by creating an account on GitHub.

#

everything here is used

#

and I have no fucking idea what to remove

#

and what to move

flint idol Sep 3, 2024, 11:44 PM

#

flint idol https://github.com/OBOS-dev/obos/blob/master/src/oboskrnl/mm/page.h#L18

any help on how to unfuck this is much appreciated

flint idol Sep 4, 2024, 12:56 AM

#

might go for a bit to read wininternals

#

and books about NT design

#

looks fun to read lapfedmoment

#

dam

#

why it so expensive O_O

#

would cost me $150 for part1+2

#

maybe I should find older things

lean glen Sep 4, 2024, 1:13 AM

#

flint idol looks fun to read <:lapfedmoment:815518338783641600>

filetype:pdf

#

lol

flint idol Sep 4, 2024, 1:18 AM

#

piracy, fun

#

microsoft has enough money anyway

lean glen Sep 4, 2024, 1:45 AM

#

sorry im not gonna pay 150$ for a pdf

flint idol Sep 4, 2024, 1:45 AM

#

fr

lean glen Sep 4, 2024, 1:46 AM

#

Even worse

#

You're paying 150$ for a book describing the 200$ software you bought

flint idol Sep 4, 2024, 1:46 AM

#

who even buys windows

#

if it don't come with your computer

#

just don't activate it

#

what're they gonna do about it

#

make me not be able to set a wallpaper?

lean glen Sep 4, 2024, 1:51 AM

#

Nah you can set a wallpaper just not through settings

#

Found this out when I was 12 at my friend's house

#

He didn't activate windows so I tried right clicking an image and click on set as wallpaper and voila

flint idol Sep 4, 2024, 1:57 AM

#

lol

#

so like basically nothing happens

#

except for ms meatriders attacking you when you send a screenshot of your desktop

#

and it says "Active windows"

lean glen Sep 4, 2024, 2:00 AM

#

Yeah the only annoying thing is the watermark

#

But why pay when there's stuff like massgrave

ornate ginkgo Sep 4, 2024, 5:52 AM

#

Well you usually pay because that's the law, whether you agree with it or not

vernal chasm Sep 4, 2024, 6:39 AM

#

ornate ginkgo Well you usually pay because that's the law, whether you agree with it or not

Microsoft doesn't really care if you pay or not

white mulch Sep 4, 2024, 6:42 AM

#

you can also just transfer the license from an old pc that you bought

vernal chasm Sep 4, 2024, 6:42 AM

#

Or that

#

You can also get a key for like 5 bucks...

ornate ginkgo Sep 4, 2024, 7:09 AM

#

vernal chasm Microsoft doesn't really care if you pay or not

My point wasn't about microsoft getting paid.

inland radish Sep 4, 2024, 9:39 AM

#

ornate ginkgo Well you usually pay because that's the law, whether you agree with it or not

I've only seen one entity in my country pay for windows

#

which is schools

#

not a single individual that I've seen has paid for windows

empty kernel Sep 4, 2024, 9:47 AM

#

In my state within my country, the education department has some kind of arrangement with Microsoft so all students and teachers in public schools get free office 365, minecraft education and windows 10 education on school devices.

inland radish Sep 4, 2024, 9:49 AM

#

We had free web Office too while I was in high school

haughty abyss Sep 4, 2024, 10:10 AM

#

its just MS doing adobe moment

#

get your sw everywhere so it becomes the standard

#

and then people use it when they grow up

empty kernel Sep 4, 2024, 10:20 AM

#

haughty abyss its just MS doing adobe moment

My school gave all the students free adobe licences last year.

lunar narwhal Sep 5, 2024, 2:54 PM

#

flint idol *microsoft has enough money anyway*

what about the authors of the book

flint idol Sep 5, 2024, 11:34 PM

#

might be stopping osdev for quite a bit

#

(this can range from a day to months)

#

but until then....

#

I will be playing mc or doing homework

#

(or maybe doing dev for the 6502)

#

might still be active here

#

my thread won't be tho

vernal chasm Sep 6, 2024, 4:02 AM

#

flint idol might be stopping osdev for quite a bit

sad

real pecan Sep 6, 2024, 7:55 AM

#

flint idol might be stopping osdev for quite a bit

what happened

flint idol Sep 6, 2024, 9:52 AM

#

real pecan what happened

I'm getting pretty tired after school and simply cannot find the time

real pecan Sep 6, 2024, 10:33 AM

#

i see

#

well i hope u come back and dont end up like marca

flint idol Sep 6, 2024, 11:44 AM

#

real pecan well i hope u come back and dont end up like marca

I probably will when I get time

flint idol Sep 6, 2024, 8:30 PM

#

I have some time rn

flint idol Sep 6, 2024, 8:31 PM

#

flint idol https://github.com/OBOS-dev/obos/blob/master/src/oboskrnl/mm/page.h#L18

and I have found a solution for this

#

I'll explain it in a bit

real pecan Sep 6, 2024, 8:50 PM

#

nice

flint idol Sep 6, 2024, 8:54 PM

#

flint idol I'll explain it in a bit

SO BASICALLY

#

I replace this with a range of pages

#

idk what kinda data structure I'll use to store the page ranges tho

#

theoretically as long as the regions do not overlap, I can use an rb-tree

flint idol Sep 6, 2024, 8:55 PM

#

flint idol I replace this with a range of pages

then to swap in/out

#

I take in a region pointer and a offset

#

the standby and dirty list will be lists of:

struct page /* TODO: name */
{
    uintptr_t virt;
    uintptr_t phys;
    bool huge_page;
};```

flint idol Sep 6, 2024, 8:59 PM

#

flint idol I take in a region pointer and a offset

to page out, I simply check if the page was modified since the last swap out

#

if so I setup a struct page

#

and slap it onto the dirty list

#

otherwise I also setup a struct page

#

but instead slap it onto the standby list

#

to swap in

#

I check if the page is on standby or is dirty

#

if so I can remove it from said list

#

and remap it with the same phys. address

#

otherwise I gotta go through the entire dance of reading it from the swap medium

#

I'll also have some sort of virtual page info struct

#

which contains prot bits

#

and redesign the arch-specific api to query/set page stuff

flint idol Sep 7, 2024, 1:09 AM

#

#

guess we're doin chatgpt now /j

vale nymph Sep 7, 2024, 1:11 AM

#

flint idol

add chatgpt system calls, say obos has deep llm integration in the kernel, print vc money, get rich, retire, work full time on obos, profit

flint idol Sep 7, 2024, 1:12 AM

#

vale nymph add chatgpt system calls, say obos has deep llm integration in the kernel, print...

genius

#

obos will have the best kernel

thick jolt Sep 7, 2024, 8:27 AM

#

flint idol I want to refrain from the latter

please dont rewrite the kernel

#

i actually wanna see OBOS finished

#

😭

flint idol Sep 7, 2024, 11:21 AM

#

thick jolt please dont rewrite the kernel

I won't

thick jolt Sep 7, 2024, 11:21 AM

#

flint idol I won't

thank you

flint idol Sep 7, 2024, 11:51 AM

#

flint idol SO BASICALLY

time to start this

#

nooo

#

may god have mercy on my soul

#

flint idol Sep 7, 2024, 12:18 PM

#

I need to implement pg_cmp_pages to compare two page ranges

#

thinkong

#

inline static int pg_cmp_pages(const page_range* left, const page_range* right)
{
}```

#

well if left->virt < right->virt

#

then it returns -1

#

if it's greater it returns 1 IF !in_range(left->virt, left->virt+left->size, right->virt)

#

otherwise zero

#

#define in_range(ra,rb,x) (((x) >= (ra)) && ((x) < (rb)))
inline static int pg_cmp_pages(const page_range* left, const page_range* right)
{
    if (left->virt < right->virt)
        return -1;
    if (left->virt > right->virt)
        return (!in_range(left->virt, left->virt+left->size, right->virt)) ? 1 : 0;
    return 0;
}
#undef in_range```

lean glen Sep 7, 2024, 12:38 PM

#

flint idol ```c #define in_range(ra,rb,x) (((x) >= (ra)) && ((x) < (rb))) inline static int...

yk you could just

#

return !in_range(whatever)

flint idol Sep 7, 2024, 12:39 PM

#

you cannot for rb trees

lean glen Sep 7, 2024, 12:39 PM

#

what

flint idol Sep 7, 2024, 12:39 PM

#

it needs the distinction between less and greater

lean glen Sep 7, 2024, 12:39 PM

#

you convert a bool to an int

#

in the ternary statement

#

you can literally just return the bool

flint idol Sep 7, 2024, 12:39 PM

#

wait I thought you meant

#

for the entire function

#

inline static int pg_cmp_pages(const page_range* left, const page_range* right)
{
    return in_range(left->virt, left->virt+left->size, right->virt);
}```

flint idol Sep 7, 2024, 12:40 PM

#

flint idol ```c inline static int pg_cmp_pages(const page_range* left, const page_range* ri...

I thought you meant I should do this

inland radish Sep 7, 2024, 12:43 PM

#

flint idol I need to implement `pg_cmp_pages` to compare two page ranges

what for?

flint idol Sep 7, 2024, 12:44 PM

#

well I'm changing this giant struct to represent a range of pages

#

and I'm using an rb-tree to store the ranges

real pecan Sep 7, 2024, 12:48 PM

#

so ur making folios or is it still virtual memory related

flint idol Sep 7, 2024, 12:49 PM

#

real pecan so ur making folios or is it still virtual memory related

what's folios

real pecan Sep 7, 2024, 12:49 PM

#

https://www.alibabacloud.com/blog/new-features-of-linux-memory-management---memory-folios_600565

flint idol Sep 7, 2024, 12:50 PM

#

real pecan so ur making folios or is it still virtual memory related

still vmem related

real pecan Sep 7, 2024, 12:50 PM

#

f

#

wait until u learn about vm_area

#

your struct page is basically this https://litux.nl/mirror/kerneldevelopment/0672327201/ch14lev1sec2.html

devout niche Sep 7, 2024, 12:53 PM

#

flint idol well I'm changing [this giant struct](https://github.com/OBOS-dev/obos/blob/mast...

how many bytes is that, about 128?

flint idol Sep 7, 2024, 12:53 PM

#

yes

flint idol Sep 7, 2024, 12:53 PM

#

real pecan your struct page is basically this https://litux.nl/mirror/kerneldevelopment/067...

yeah, except it's for a single page

real pecan Sep 7, 2024, 12:53 PM

#

devout niche how many bytes is that, about 128?

and this exists for every virtually mapped page

devout niche Sep 7, 2024, 12:53 PM

#

i have 64 (well, 56, but padded on 64-bit platforms; 32 on 32-bit platforms)

real pecan Sep 7, 2024, 12:53 PM

#

nah your struct page is for physical pages

#

this is a virtual page descriptor

devout niche Sep 7, 2024, 12:54 PM

#

oh, i see it now

#

the pagedOut bit is a giveaway

flint idol Sep 7, 2024, 12:56 PM

#

and pageable

#

and isPrivateMapping

real pecan Sep 7, 2024, 12:57 PM

#

Lol

flint idol Sep 7, 2024, 12:57 PM

#

I think this entire thing screams "virtual page"

#

except for the name

real pecan Sep 7, 2024, 12:57 PM

#

Me when my entire ram is consumed by vmm overhead

lean glen Sep 7, 2024, 12:58 PM

#

flint idol I think this entire thing screams "virtual page"

I feel like you could put a bunch of those fields inside unions

devout niche Sep 7, 2024, 12:58 PM

#

in my case the division of labour for that virtual page struct is between the physical page frame struct, the working set list entry, and the vm map entry

lean glen Sep 7, 2024, 12:58 PM

#

yeah that's essentially what I do too

flint idol Sep 7, 2024, 12:58 PM

#

anyway

#

I must go back to de-shittifying this

lean glen Sep 7, 2024, 12:58 PM

#

I think it'd be on a per-page-in-anon basis maybe?

#

idk

real pecan Sep 7, 2024, 12:59 PM

#

devout niche in my case the division of labour for that virtual page struct is between the ph...

so whats the approximate byte overhead for a single anon page mapping

devout niche Sep 7, 2024, 1:00 PM

#

real pecan so whats the approximate byte overhead for a single anon page mapping

worst case is 64 bytes on the page struct, 1 page on the working set list, 64? bytes on the map entry, well, i could go on for a while with the constant overheads

real pecan Sep 7, 2024, 1:01 PM

#

i see

devout niche Sep 7, 2024, 1:02 PM

#

but if you exclude the overheads that are dominated down to being fairly negligible when there's lots of pages mapped, it's a word (a single working set entry)

#

i am excluding the cost of page tables, prototype page tables, map entry struct, the physical vm_page struct, etc, those can dominate if you, say, make a hundred thousand memory objects and map a single page of each, each 1gib apart in the address space

real pecan Sep 7, 2024, 1:04 PM

#

whats a prototype page table

devout niche Sep 7, 2024, 1:05 PM

#

an abstract page table like data structure that keeps track of pages for files, anonymous shared vm objects, those are organised as a 4-level table, and (in the form of a single pointer coupled with a refcount) forked pages

real pecan Sep 7, 2024, 1:06 PM

#

tahts interesting

#

is that some common concept or did you invent that yourself

devout niche Sep 7, 2024, 1:07 PM

#

the 4-level tables are excessive and i have a plan to make it so that you don't have to pay 4 pages if you just want to e.g. mmap the very first page of a file

#

my plan is probably to make it so that the 4 levels can be collapsed to however many are actually needed to map the pages that are actually mapped of a file

real pecan Sep 7, 2024, 1:08 PM

#

Do you know how other kernels solve this?

devout niche Sep 7, 2024, 1:08 PM

#

real pecan is that some common concept or did you invent that yourself

it is basically just a particular way of implementing something that first started in mach with its concept of memory objects

real pecan Sep 7, 2024, 1:09 PM

#

I see

flint idol Sep 7, 2024, 1:09 PM

#

I think I'm ditching the kernel gdbstub

#

it's pretty slow and useless

devout niche Sep 7, 2024, 1:10 PM

#

other popular data structures have been a hash table of {vnode, page} (e.g. sunos vmm), linux also uses radix trees for this, netbsd uses a wAVL tree for file pages and a 2-level table with variably-sized root-level for files, windows uses linear tables allocated from the paged pool

real pecan Sep 7, 2024, 1:10 PM

#

Interesting

devout niche Sep 7, 2024, 1:11 PM

#

well nowadays i think they might allocate linear spans on windows and link them into an avl tree because the linear tables are no good if you want to map only a few pages but they're 64gb into the file

devout niche Sep 7, 2024, 1:13 PM

#

flint idol I think I'm ditching the kernel gdbstub

i think they're useful to have when testing on real hardware

#

so i wasted some time implementing polled send/receive for e1000 in case i need to debug my kernel on my laptop

flint idol Sep 7, 2024, 1:14 PM

#

until I find the time to fix it

devout niche Sep 7, 2024, 1:14 PM

#

if any documentation exists for whatever tf the raspi4 has (let's be honest, there won't be any documentation, documentation is for computers with real cpus, i.e. amd64 cpus) then that's another one i should write

flint idol Sep 7, 2024, 1:14 PM

#

flint idol until I find the time to fix it

it's not going to be used

thick jolt Sep 7, 2024, 1:19 PM

#

flint idol

page

#

page.

#

PAGE FAULT!

#

that gives you nightmares!

#

ksan violation!

flint idol Sep 7, 2024, 1:20 PM

#

you're giving me flashbacks

thick jolt Sep 7, 2024, 1:20 PM

#

hahaha

flint idol Sep 7, 2024, 4:22 PM

#

wait

#

if I have a range of pages

#

struct page_range

#

how the fuck

#

do I figure out

#

how many working sets each page is in

#

or the pages' ages

#

without splitting the regions and updating the appropriate fields

#

whenever a page gets paged out

#

or is put in a working set

#

one thing I could do is.........

#

no idea

#

I can't do it with arch-specific hooks

#

like I can't have something like void Mm_PageAge(uintptr_t addr, uint8_t* age, bool rw);

#

at least not efficiently on archs that can't don't have 8 os-reserved bits in a PTE needed to store that kinda stuff

devout niche Sep 7, 2024, 4:30 PM

#

flint idol at least not efficiently on archs that can't don't have 8 os-reserved bits in a ...

i would be storing it in working set list entries

flint idol Sep 7, 2024, 4:31 PM

#

ok

#

gave me an idea

#

each working set list entry for a page is the same thing

#

and they will be stored in a non-intrusive linked list

#

oh god

#

the error log is ~2k lines

#

yeah this might call for a vmm rewrite

#

no public api of the vmm will be changed

devout niche Sep 7, 2024, 5:30 PM

#

flint idol yeah this might call for a vmm rewrite

i know the feeling

#

i am deep into 9000 lines of (very dense) code in the keyronex vmm and worry how will i ever make big changes to it (like the fine locking plan i have eventually)

limber wave Sep 7, 2024, 7:53 PM

#

flint idol yeah this might call for a vmm rewrite

how many times by now?!

#

I am starting to think that you secretly are sadistic and enjoy giving yourself pain

flint idol Sep 7, 2024, 7:54 PM

#

limber wave how many times by now?!

this would be the third revision of the vmm

limber wave Sep 7, 2024, 7:55 PM

#

yeah

#

and the pmm?

flint idol Sep 7, 2024, 7:55 PM

#

has stayed the same since forever

#

since my 2nd rewrite of the kernel

#

or 3rd

limber wave Sep 7, 2024, 7:55 PM

#

ahhh

flint idol Sep 7, 2024, 7:55 PM

#

(this is the fifth rewrite btw)

#

(fourth was a disaster)

#

(third was just not designed well)

#

(2nd/1st were shit)

limber wave Sep 7, 2024, 7:56 PM

#

I suggest to start an entire channel in here just for people to try out your curse, and maybe help you

limber wave Sep 7, 2024, 7:56 PM

#

flint idol (this is the fifth rewrite btw)

hah

#

I have rewritten my bootloader like 20 times

flint idol Sep 7, 2024, 7:56 PM

#

lol

limber wave Sep 7, 2024, 7:56 PM

#

and never went further than that

#

because I enjoy bootloaders

flint idol Sep 7, 2024, 7:57 PM

#

.!t limine

umbral iceBOT Sep 7, 2024, 7:57 PM

#

https://github.com/limine-bootloader/limine-c-template

GitHub

GitHub - limine-bootloader/limine-c-template: A simple template for...

A simple template for building a Limine-compliant kernel in C. - limine-bootloader/limine-c-template

flint idol Sep 7, 2024, 7:57 PM

#

oh nvm

#

you chose this pain

limber wave Sep 7, 2024, 7:57 PM

#

yessir

flint idol Sep 7, 2024, 7:57 PM

#

#bootloaders message

#

well gl with your btldr

#

after vmm rewrite is TTYs

#

so in a couple weeks bcuz school

limber wave Sep 7, 2024, 7:59 PM

#

flint idol so in a couple weeks bcuz school

I just skipped school because I liked it so much

#

modern problems require modern (bad) solutions

#

please kids don't try this at home

flint idol Sep 7, 2024, 7:59 PM

#

wouldn't recommend

#

(my parents would kill me)

limber wave Sep 7, 2024, 7:59 PM

#

flint idol wouldn't recommend

only reason so for me is that I miss being at school with all the people

flint idol Sep 7, 2024, 10:24 PM

#

so I might actaully make a driver interface in the form of a library, sounds fun

#

oDriverinterface

#

comes with:

a dynamic loader
a lot of interface functions
maybe even pnp
the oberrow curse

limber wave Sep 7, 2024, 10:46 PM

#

you are quite wrong here

#

you see, everything is optional, except the last in the list

flint idol Sep 7, 2024, 10:52 PM

#

flint idol comes with: - a dynamic loader - a lot of interface functions - maybe even pnp -...

the loader will support x86-64 relocations for now

#

maybe m68k soon

#

and x86

#

since no one uses msvc I'll be forcing gcc

#

extensions

limber wave Sep 7, 2024, 10:56 PM

#

flint idol since no one uses msvc I'll be forcing gcc

does msvc even support m68k?

flint idol Sep 7, 2024, 10:57 PM

#

doubt

#

each driver will have a rw callback

#

along with ioctl

#

as well as a cleanup cb

#

for unload

#

and idk how fs drivers will work

#

for making, moving, and removing files

#

I guess I can just

#

have cbs for those

#

they'll take in a dirent

#

and then I'll also have a probe cb

#

which probes a partition to see if there is an fs there

flint idol Sep 7, 2024, 11:02 PM

#

flint idol for making, moving, and removing files

and there will also be chown and chmod cbs which take in vnodes

lean glen Sep 7, 2024, 11:59 PM

#

what the fuck is a CB

flint idol Sep 7, 2024, 11:59 PM

#

callback

#

or centerback in soccer

lean glen Sep 8, 2024, 12:00 AM

#

ah

real pecan Sep 8, 2024, 12:25 AM

#

lean glen what the fuck is a CB

CBT without the T

flint idol Sep 8, 2024, 8:03 PM

#

I hate this nooo

#

a full rewrite of the vmm

#

is going to be pain

#

commit msg will be:
Mm: Small changes

#

then continue to have 2k+ deletions

#

and 2k+ additions

vale nymph Sep 8, 2024, 8:13 PM

#

lgtm

flint idol Sep 8, 2024, 8:45 PM

#

I hate this

#

I hate this

#

I hate this

#

I no longer hate this as much

flint idol Sep 8, 2024, 9:27 PM

#

flint idol or the pages' ages

but how would I figure out the swap allocation id for a swapped out page

#

when it's being swapped in once again

flint idol Sep 8, 2024, 9:28 PM

#

flint idol but how would I figure out the swap allocation id for a swapped out page

(basically that thing just represents allocated space in the swapfile)

#

uhhhhhh

#

maybe

#

in the page region thingy

#

I have a list of paged out pages

#

something like this:

typedef struct swap_range
{
    uintptr_t id;
    uintptr_t virt_base;
    size_t    size;
} swap_range;```

#

but since most pages are going to be paged out

#

that defeats the entire purpose of my vmm rewrite

#

yeah I haven't a clue

flint idol Sep 10, 2024, 1:03 AM

#

I got an idea

#

to do that

#

I just need to completely re-design how my swap works

#

obos_status(* swap_resv)(struct swap_device* dev, uintptr_t *id, size_t nPages);
obos_status(* swap_free)(struct swap_device* dev, uintptr_t  id, size_t nPages, size_t offsetBytes);
obos_status(*swap_write)(struct swap_device* dev, uintptr_t  id, uintptr_t phys, size_t nPages, size_t offsetBytes);
obos_status(* swap_read)(struct swap_device* dev, uintptr_t  id, uintptr_t phys, size_t nPages, size_t offsetBytes);```

#

for example here

#

instead of swap_resv giving me an id

#

it just takes in a virtual address as the handle

#

and the swap device somehow (hashmap, fancy tree?) stores a virtual memory -> place in swap file

#

prob. hashmap on disk

#

and a tree for the in-ram swap

#

obos_status(* swap_resv)(struct swap_device* dev, uintptr_t virt, bool huge_page);
obos_status(* swap_free)(struct swap_device* dev, uintptr_t virt, bool huge_page);
obos_status(*swap_write)(struct swap_device* dev, uintptr_t virt /* must be mapped */, size_t nPages);
obos_status(* swap_read)(struct swap_device* dev, uintptr_t virt /* must be mapped */, size_t nPages);```

rapid cairn Sep 10, 2024, 1:11 AM

#

hi

flint idol Sep 10, 2024, 1:11 AM

#

hello

flint idol Sep 10, 2024, 1:11 AM

#

flint idol I just need to completely re-design how my swap works

so this would become

rapid cairn Sep 10, 2024, 1:11 AM

#

what are you up to?

flint idol Sep 10, 2024, 1:11 AM

#

flint idol ```c obos_status(* swap_resv)(struct swap_device* dev, uintptr_t virt, bool huge...

that

flint idol Sep 10, 2024, 1:11 AM

#

rapid cairn what are you up to?

fixing my vmm

rapid cairn Sep 10, 2024, 1:11 AM

#

whats wrong with it?

flint idol Sep 10, 2024, 1:11 AM

#

currently there's a big design flaw

#

where each virtual page

#

is stored in a 128-byte struct

#

causing very large vmm overhead

rapid cairn Sep 10, 2024, 1:11 AM

#

how much overhead we talkin?

flint idol Sep 10, 2024, 1:11 AM

#

let's see

#

I'll do some math to see how much 1G of memory allocated would be

rapid cairn Sep 10, 2024, 1:12 AM

#

also do you dev on windows or linux?

flint idol Sep 10, 2024, 1:12 AM

#

used to dev on windows

#

then switched to linux mid-june

rapid cairn Sep 10, 2024, 1:12 AM

#

yay a fellow linux freedom figher

flint idol Sep 10, 2024, 1:12 AM

#

flint idol I'll do some math to see how much 1G of memory allocated would be

8M overhead

#

for 1G allocated

rapid cairn Sep 10, 2024, 1:13 AM

#

8 megabytes, thats a LOT, so is this your malloc and free stuff?

flint idol Sep 10, 2024, 1:13 AM

#

you could say that I guess

#

basically it's the thing that deals with allocating and managing virtual memory "pages"

#

e.g., mapping files, swapping out stuff, mapping anon memory

rapid cairn Sep 10, 2024, 1:14 AM

#

OH so this is mapping psyical memory to virtual right?

flint idol Sep 10, 2024, 1:14 AM

#

flint idol e.g., mapping files, swapping out stuff, mapping anon memory

and many other duties

flint idol Sep 10, 2024, 1:14 AM

#

rapid cairn OH so this is mapping psyical memory to virtual right?

as the very base level, yes

rapid cairn Sep 10, 2024, 1:14 AM

#

and it has a big overhead due to what?

flint idol Sep 10, 2024, 1:14 AM

#

it must store info about a page

#

for example the offset in the swap file

#

that it lies in

#

or the file it's mapped to

rapid cairn Sep 10, 2024, 1:15 AM

#

so what part of the code do you suspect is causing issues?

flint idol Sep 10, 2024, 1:15 AM

#

https://github.com/OBOS-dev/obos/blob/master/src/oboskrnl/mm/page.h#L18

GitHub

obos/src/oboskrnl/mm/page.h at master · OBOS-dev/obos

I was bored... Contribute to OBOS-dev/obos development by creating an account on GitHub.

#

but the thing is, the entire thing depends on that one struct

rapid cairn Sep 10, 2024, 1:16 AM

#

have you tried chatgpt /j

flint idol Sep 10, 2024, 1:16 AM

#

flint idol but the thing is, the entire thing depends on that one struct

so I need to basically rewrite everything

rapid cairn Sep 10, 2024, 1:16 AM

#

oh, well at that porint just reimplement paging

flint idol Sep 10, 2024, 1:16 AM

#

rapid cairn have you tried chatgpt /j

chatgpt stoobid

#

that's many many work

#

to rewrite all

#

in the vmm

rapid cairn Sep 10, 2024, 1:17 AM

#

well not ALL but more like the struct, so something in it is causing an 8MB overhead, correct?

flint idol Sep 10, 2024, 1:17 AM

#

and a big misconception in beginners is that the VMM just deals with mapping pages

flint idol Sep 10, 2024, 1:17 AM

#

rapid cairn well not ALL but more like the struct, so something in it is causing an 8MB over...

that's if you allocated 1G

#

for each mapped page, it's about a 128-byte overhead

rapid cairn Sep 10, 2024, 1:17 AM

#

and within that it grows I suppose?

flint idol Sep 10, 2024, 1:17 AM

#

wdym?

rapid cairn Sep 10, 2024, 1:18 AM

#

I mean the overhead grows

flint idol Sep 10, 2024, 1:18 AM

#

yes

#

1 mapped page would be 128-bytes

#

2 would be 256 bytes

#

so on and so forth

rapid cairn Sep 10, 2024, 1:18 AM

#

that 128 byte overhead (not much) grows into a WAY bigger 8MB overhead which would grow to become a bigger and bigger issue

flint idol Sep 10, 2024, 1:18 AM

#

indeed

#

just figured out I did my math wrong

rapid cairn Sep 10, 2024, 1:19 AM

#

is it not as big of an overhead or is it way bigger?

flint idol Sep 10, 2024, 1:19 AM

#

a lot smaller

#

2048-bytes

#

but still unwanted

#

as it grows and grows

rapid cairn Sep 10, 2024, 1:20 AM

#

still thats WAY better than 8MB, I mean windows has some overhead too lemme check it

flint idol Sep 10, 2024, 1:20 AM

#

certainly not as much overhead as this

rapid cairn Sep 10, 2024, 1:21 AM

#

true, but this is FAR better than 8MB, I mean at least this is ACCEPTABLE I mean its still a good chunck of overhead but still

flint idol Sep 10, 2024, 1:22 AM

#

but with the solution of simply having a range of pages

#

that gets cut down to a static

#

lemme see how much

#

104 bytes overhead

#

for each range of pages allocated

#

say you allocated 1 TiB

#

that would be 104 bytes overhead

#

say you allocated a page

rapid cairn Sep 10, 2024, 1:23 AM

#

104 byte overhead with 1 TERABYTE of ram

flint idol Sep 10, 2024, 1:23 AM

#

104 bytes overhead

flint idol Sep 10, 2024, 1:23 AM

#

rapid cairn 104 byte overhead with 1 TERABYTE of ram

more like virtual address space

#

the kernel doesn't hand out ram like it's nothin'

rapid cairn Sep 10, 2024, 1:23 AM

#

flint idol more like virtual address space

okay that makes more sense lol

flint idol Sep 10, 2024, 1:24 AM

#

the program thinks it gets 1 TiB

#

but in reality

#

the kernel doesn't "commit" the memory until it's touched

rapid cairn Sep 10, 2024, 1:24 AM

#

so its not actually used until its asked for

flint idol Sep 10, 2024, 1:24 AM

#

indeed

#

you can also reserve virtual address space

#

if you do that

#

it's unmapped

#

but nothing can reallocate it

rapid cairn Sep 10, 2024, 1:25 AM

#

what about stuff like malloc? how would it work then, for me it just makes a 1MB single page, then splits it to store data as requested then free marks it as free and merges it, so how does it work for you?

flint idol Sep 10, 2024, 1:26 AM

#

my kernel's allocator (for general structs, not pages/physical pages) simply has a free list and allocated list

#

when you allocate something

#

it finds a "region" with sufficient memory

#

if no such thing exists it makes a new one with sufficient size

#

then it takes some memory from a node in the free list

#

then it throws at you that ptr

#

on free what it does is it takes your ptr

#

if it's the only allocated region it gets purged (freed)

#

otherwise it gets added to the free list

#

https://github.com/OBOS-dev/obos/blob/master/src/oboskrnl/allocators/basic_allocator.c

GitHub

obos/src/oboskrnl/allocators/basic_allocator.c at master · OBOS-dev...

I was bored... Contribute to OBOS-dev/obos development by creating an account on GitHub.

#

wrote this code months ago

rapid cairn Sep 10, 2024, 1:29 AM

#

for me I just pass free the pointer and it does its magic

flint idol Sep 10, 2024, 1:29 AM

#

yeah same

#

-# I probably forget to free a bunch of stuff

#

-# 20M of memory committed? more like 20M memory leaked

rapid cairn Sep 10, 2024, 1:30 AM

#

for me the kernel API is also the userspace API lol, since its all going to be in kernel mode because im dumb

flint idol Sep 10, 2024, 1:30 AM

#

won't go well

#

but you do you

rapid cairn Sep 10, 2024, 1:30 AM

#

well its more ring 0 lol, the Kernel API will be private then it will have public stuff for the userspace to use

#

it will kinda be like templeOS, just no religion

flint idol Sep 10, 2024, 1:31 AM

#

not quite a userspace if it's in ring 0 lol

#

since a program (with or without malicious intent) can call a kernel mode function without you knowing it

rapid cairn Sep 10, 2024, 1:32 AM

#

true, on the other hand, who can hack an OS when they cannot even understand the API!

flint idol Sep 10, 2024, 1:32 AM

#

lmao

rapid cairn Sep 10, 2024, 1:32 AM

#

not even the dev understands the API, thats how bad my code is

flint idol Sep 10, 2024, 1:32 AM

#

only thing that runs in kernel mode in my kernel is the kernel ofc and the kernel modules (drivers)

rapid cairn Sep 10, 2024, 1:33 AM

#

so you have a driver system already?

flint idol Sep 10, 2024, 1:33 AM

#

yes

rapid cairn Sep 10, 2024, 1:33 AM

#

what about apps?

flint idol Sep 10, 2024, 1:33 AM

#

also planning on making a library for that

flint idol Sep 10, 2024, 1:33 AM

#

rapid cairn what about apps?

hehe userspace

#

you're funny

rapid cairn Sep 10, 2024, 1:33 AM

#

not there yet, or working on it?

flint idol Sep 10, 2024, 1:33 AM

#

not there yet

#

(need to rewrite that vmm)

rapid cairn Sep 10, 2024, 1:34 AM

#

are the kernel modules loaded from disk?

flint idol Sep 10, 2024, 1:34 AM

#

they can be

rapid cairn Sep 10, 2024, 1:34 AM

#

so it works?

flint idol Sep 10, 2024, 1:34 AM

#

yes

rapid cairn Sep 10, 2024, 1:34 AM

#

so how do kernel modules work?

flint idol Sep 10, 2024, 1:35 AM

#

so basically

#

a kernel module is a dynamically loaded elf file

#

kinda like a dynamic library

#

that's linked to the kernel at run-time

rapid cairn Sep 10, 2024, 1:35 AM

#

so its a program more or less?

flint idol Sep 10, 2024, 1:35 AM

#

when it's loaded

flint idol Sep 10, 2024, 1:35 AM

#

rapid cairn so its a program more or less?

you could say that

#

it has its own header alongside the elf header which can be anywhere in the header, or optionally in a section to speed up search

#

said header has info on the kernel module

#

like how pnp should find it

rapid cairn Sep 10, 2024, 1:36 AM

#

flint idol that's linked to the kernel at run-time

kinda like templeOS, loaded at run time, but in this case not compiled at run time

flint idol Sep 10, 2024, 1:36 AM

#

and what its name is (optional)

#

and also a table of functions for the kernel to call when it wants somethin from the driver

flint idol Sep 10, 2024, 1:37 AM

#

rapid cairn kinda like templeOS, loaded at run time, but in this case not compiled at run ti...

well idk why it'd be compiled at runtime

rapid cairn Sep 10, 2024, 1:37 AM

#

flint idol well idk why it'd be compiled at runtime

in temple OS the entire OS is just JIT

flint idol Sep 10, 2024, 1:37 AM

#

concern

rapid cairn Sep 10, 2024, 1:37 AM

#

other than the bootloader and the compiler lol

#

https://templeos.org/

#

welp, gn

flint idol Sep 10, 2024, 1:39 AM

#

gn

flint idol Sep 10, 2024, 1:56 AM

#

damn obos has been cursed since the start

#

most stable obos code

📎 bootloader.asm

flint idol Sep 10, 2024, 2:05 AM

#

flint idol ```c obos_status(* swap_resv)(struct swap_device* dev, uintptr_t virt, bool huge...

marker

flint idol Sep 10, 2024, 2:05 AM

#

flint idol and a big misconception in beginners is that the VMM just deals with mapping pag...

depending on your goals this could be all it does

thick jolt Sep 10, 2024, 4:20 AM

#

flint idol then switched to linux mid-june

welcome to linux land 😎

limber wave Sep 10, 2024, 6:55 PM

#

flint idol most stable obos code

it's using bits 64 instead of 16

flint idol Sep 10, 2024, 7:08 PM

#

limber wave it's using bits 64 instead of 16

Ik

#

I made this a year ago

hollow geyser Sep 10, 2024, 11:50 PM

#

make it 4 bit

#

so i can run it on my pc

flint idol Sep 10, 2024, 11:50 PM

#

Ok

hollow geyser Sep 10, 2024, 11:50 PM

#

(i have an intel 4004 as a cpu)

thick jolt Sep 11, 2024, 5:30 AM

#

flint idol Ok

Ok

flint idol Sep 11, 2024, 8:20 PM

#

Will work on this some more in a tiny bit

flint idol Sep 11, 2024, 10:37 PM

#

the pf handler code is gonna be pain to change 😭

#

😭

#

I may as well rewrite that file

#

if I'm being completely honest

#

the kernel will not work when I get it to finally compile

flint idol Sep 11, 2024, 10:50 PM

#

flint idol the kernel will not work when I get it to finally compile

first thing it should do is page fault executing nullptr

#

it's gottne to compiling drivers

#

*gotten

#

it's linking

#

ok it linked

#

and as intended

#

it crashes

#

or maybe not as intended

#

the curse is techincally back

#

but not really

#

since nothing is supposed to work

#

intentionally

vale nymph Sep 11, 2024, 11:03 PM

#

obos without the oberrow curse is like an angel without its wings

flint idol Sep 11, 2024, 11:03 PM

#

#

now that's a bug unfortunately

#

and not a result of unimplemented stuff

#

since freeing doesn't exist

#

I can do some stuff

#

bug in MmH_FindAvailableAddress

#

something tells me

#

that I don't register the regions

#

I don't

#

inline static int pg_cmp_pages(const page_range* left, const page_range* right)
{
    if (left->virt < right->virt)
        return -1;
    if (left->virt > right->virt)
        return in_range(left->virt, left->virt+left->size, right->virt);
    return 0;
}```

#

that's buggy

#

idk how

#

as in

#

idk how I shoulda made it in the first place

#

I feel like fadanoid would know

#

"m68k"

#

it'll work eventually

#

(as in saying m68k to summon fadanoid)

#

((not the code))

#

(((that ain't workin for a while)))

flint idol Sep 11, 2024, 11:17 PM

#

flint idol ```c inline static int pg_cmp_pages(const page_range* left, const page_range* ri...

hmm

#

I think I should just return a memcmp

#

or smth like that

limber wave Sep 11, 2024, 11:25 PM

#

flint idol "m68k"

hmm

flint idol Sep 11, 2024, 11:42 PM

#

it's gotten past uACPI init

#

and it crashes

#

at VFS init

#

because it tries to politely ask the initrd driver

#

for some files

#

but it pfs because the memory protection thing is unimplemented

flint idol Sep 12, 2024, 12:55 AM

#

the freeing function should be implemented for the most part

flint idol Sep 12, 2024, 1:21 AM

#

nooo

#

Curse is bacl

#

*back'

#

ok

#

it only exists on reboot

#

other than the fact that I temporarily don't have swap support

#

and that the kernel crashes on reboot

#

this is probably nearly dobe

#

*done

flint idol Sep 12, 2024, 1:32 AM

#

flint idol other than the fact that I temporarily don't have swap support

or CoW support

#

or file mapping

#

or rather I only have one half of them

real pecan Sep 14, 2024, 11:24 AM

#

when new obos features

flint idol Sep 14, 2024, 11:24 AM

#

When I get the vmm outta the wat

#

*way

real pecan Sep 14, 2024, 11:24 AM

#

nice

flint idol Sep 14, 2024, 3:29 PM

#

flint idol When I get the vmm outta the wat

also brainfuck interpreter

thick jolt Sep 14, 2024, 3:30 PM

#

bro is putting a brainfuck interpreter in obos

flint idol Sep 14, 2024, 3:30 PM

#

no

#

(maybe)

#

((probably not))

thick jolt Sep 14, 2024, 3:31 PM

#

(u should) ((best))

#

(write memory manager in brainfuck) :)

flint idol Sep 14, 2024, 3:31 PM

#

write kernel in bf

thick jolt Sep 14, 2024, 3:31 PM

#

memedown

flint idol Sep 14, 2024, 3:31 PM

#

someone has to have written a gcc backend for bf

#

to output bf from C

#

or smth

#

someone did make a frontend

lean glen Sep 14, 2024, 3:55 PM

#

flint idol to output bf from C

that's insanely hard to do tho

flint idol Sep 14, 2024, 3:55 PM

#

yea I thought about it a bit

limber wave Sep 14, 2024, 4:58 PM

#

flint idol When I get the vmm outta the wat

so never? troll

flint idol Sep 14, 2024, 4:58 PM

#

no

#

soon enough

flint idol Sep 14, 2024, 10:35 PM

#

https://cdn.discordapp.com/attachments/1151528935167885363/1284643758335590516/obos.gif

flint idol Sep 14, 2024, 11:43 PM

#

fuck

#

curse back

flint idol Sep 15, 2024, 1:39 PM

#

kinda wanna try boron's firework test

#

on obos

#

https://github.com/iProgramMC/Boron/blob/master/drivers/test/source/fworktst.c#L351

GitHub

Boron/drivers/test/source/fworktst.c at master · iProgramMC/Boron

The Boron Operating System - an experimental SMP operating system inspired by a particular 1990s workstation OS - iProgramMC/Boron

#

I've found it here

#

now I will do some magic stuff to get it to work on obos

#

(there is no way this goes well)

#

I'm thinking on putting this in my test_driver

#

it was originally there just to test the elf loader for the driver interface

#

but I guess I could make it better by adding that

#

actaully

#

idk if I wanna use boron's code

#

bcuz license

#

I mean if I just never commit it

limber wave Sep 15, 2024, 2:00 PM

#

if I had to put license everytime I tested some code from other sources... well I would have more license files than source files

#

but then again, I never commit anything since I do not use github for personal projects

flint idol Sep 15, 2024, 3:11 PM

#

bruh

#

wtf is this

#

my gdb don't wanna work

#

oops

flint idol Sep 15, 2024, 4:00 PM

#

bruh

#

I hate dynamic linking

thick jolt Sep 15, 2024, 4:00 PM

#

i love dynamic linking

#

enjo

#

y

#

:)

flint idol Sep 15, 2024, 4:00 PM

#

unless

#

wait nvm

#

I was in the dynamic symbol table of the driver

#

new stuff added to the panic screen

#

namely the driver bases

thick jolt Sep 15, 2024, 4:01 PM

#

my kernel panic is better then ur kernel panic

flint idol Sep 15, 2024, 4:01 PM

#

and stack trace

thick jolt Sep 15, 2024, 4:01 PM

#

it calls u an idiot

#

:)

flint idol Sep 15, 2024, 4:02 PM

#

meme

#

#

inspiration taken from boron's panic screen

#

there is the panic on the framebuffer

#

and not from the debugcon output

#

bruh the stack trace is garbage

thick jolt Sep 15, 2024, 4:15 PM

#

lol

flint idol Sep 15, 2024, 4:16 PM

#

bro why is the test driver causing the entire kernel to crash

#

it's 5 lines

#

of code run

#

OBOS_PAGEABLE_FUNCTION void OBOS_DriverEntry(driver_id* this)
{
    this_driver = this;
    OBOS_Log("%s: Hello from test driver #1. Driver base: %p. Driver id: %d.\n", __func__, this->base, this->id);
    TestDriver_Test(this);
    OBOS_Log("Exiting from main thread.\n");
    Core_ExitCurrentThread();
}```

#

Page fault at 0xffffff0000199310 in kernel-mode while to write page at 0x0000000000000000, which is unpresent. Error code: 2```

#

that address

#

is in the test driver

#

but it isn't at the same time

short mortar Sep 15, 2024, 4:17 PM

#

Where do you get symbol names from?

flint idol Sep 15, 2024, 4:17 PM

#

the kernel has all of its symbols cached in a tree

#

rb-tree

#

so does each driver

short mortar Sep 15, 2024, 4:18 PM

#

oh because you have modules

flint idol Sep 15, 2024, 4:18 PM

#

yea

real pecan Sep 15, 2024, 4:22 PM

#

flint idol

While to write page

flint idol Sep 15, 2024, 4:22 PM

#

wait

#

for how long has that been there for

#

no wonder those pf messages always sounded kinda weird

#

I think the test driver might be cursed

#

and the stack trace

#

why are some kernel symbols resolved

#

and others aren't

flint idol Sep 16, 2024, 12:27 AM

#

old msg but good idea

#

I'll just put a kernel memory dump in a qr code

#

(not actually)

#

but maybe on panic that could be a good idea

real pecan Sep 16, 2024, 4:47 AM

#

You cant dump much in it since qr can barely store anything

rapid pulsar Sep 16, 2024, 5:07 AM

#

2953 bytes is the maximum size of a single QR code with a module size of 177×177 and low error correction.

flint idol Sep 16, 2024, 10:38 AM

#

flint idol but maybe on panic that could be a good idea

for this I meant on panic

#

to output a qr code with the panic info

haughty abyss Sep 16, 2024, 10:48 AM

#

you can still fit like

flint idol Sep 16, 2024, 10:48 AM

#

maybe a stack dump

haughty abyss Sep 16, 2024, 10:48 AM

#

kernel version + traceback + format args for panic + panic id

flint idol Sep 16, 2024, 10:48 AM

#

yeah

haughty abyss Sep 16, 2024, 10:49 AM

#

as a link to a website which reports the crash :^)

flint idol Sep 16, 2024, 10:50 AM

#

I could probably just dump the info printfed on panic to the qr code

flint idol Sep 16, 2024, 10:52 AM

#

flint idol I could probably just dump the info `printf`ed on panic to the qr code

except for the ascii art ofc

real pecan Sep 16, 2024, 10:52 AM

#

flint idol except for the ascii art ofc

No no it must be included as well

flint idol Sep 16, 2024, 10:53 AM

#

that would take 16% of the qr code's space

#

so no

real pecan Sep 16, 2024, 10:55 AM

#

Dump the entire framebuffer as qr

flint idol Sep 16, 2024, 10:55 AM

#

that'd take 1420% of the qr code

#

*142035%

vale nymph Sep 16, 2024, 10:56 AM

#

Just tell the user to take a picture and upload it to some website troll

lean glen Sep 16, 2024, 11:04 AM

#

You can use qrloops

#

Which is a bunch of QR codes that are looped

flint idol Sep 16, 2024, 11:06 AM

#

#

I added some more info to the kernel panic

flint idol Sep 16, 2024, 11:22 AM

#

made some small changes

#

waiiit

#

that crash could be because of an allocator bug

#

shit

#

TODO:

vale nymph Sep 16, 2024, 11:30 AM

#

the oberrow curse strikes again
TODO: make it not

inland radish Sep 16, 2024, 11:45 AM

#

flint idol https://github.com/iProgramMC/Boron/blob/master/drivers/test/source/fworktst.c#L...

don't worry about it

#

I won't sue, promise

inland radish Sep 16, 2024, 11:47 AM

#

flint idol the kernel has all of its symbols cached in a tree

you dont really need an rbtree unless you load symbols from the kernel file

#

i use a separate symbol array ordered by address so i can simply do a binary search

vale nymph Sep 16, 2024, 11:53 AM

#

Once I get modules working in astral the first module I should do is fireworks test

ornate ginkgo Sep 16, 2024, 2:00 PM

#

lol everyone is porting nanoshell fireworks

#

I was also thinking about porting it (or a version of it) at one point for a scheduler test

ornate ginkgo Sep 16, 2024, 2:02 PM

#

flint idol made some small changes

tbh there's a lot of noise there - how much of that is helpful in diagnosing why the system crashed?

flint idol Sep 16, 2024, 2:03 PM

#

ornate ginkgo tbh there's a lot of noise there - how much of that is helpful in diagnosing why...

Depends on where you are, on qemu most of it can be found out using gdb

#

However on real hw it's useful to have the stack trace and driver bases

#

And register dumps in the case of a cpu exception

ornate ginkgo Sep 16, 2024, 2:04 PM

#

eh i find the gpr state kind of useless

flint idol Sep 16, 2024, 2:05 PM

#

Fair

white mulch Sep 16, 2024, 2:05 PM

#

the ip, sp and cr2 sure but the other registers aren't terribly useful

ornate ginkgo Sep 16, 2024, 2:05 PM

#

yeah, thats why I specified gprs

#

maybe the stack pointer in some cases

flint idol Sep 16, 2024, 2:06 PM

#

The info about the build is there for bug reports

flint idol Sep 16, 2024, 2:07 PM

#

ornate ginkgo eh i find the gpr state kind of useless

The gpr state isn't part of the panic function itself btw

#

I just dump it in the format string for cpu exceptions

white mulch Sep 16, 2024, 2:10 PM

#

though to be fair linux also does a gpr dump KEKW

ornate ginkgo Sep 16, 2024, 2:11 PM

#

I guess thats why its called Linux and not Winux

inland radish Sep 16, 2024, 5:35 PM

#

ornate ginkgo lol everyone is porting nanoshell fireworks

i mean not really

#

nanoshell fireworks is singlethreaded

#

boron fireworks isnt

#

also fun fact, I ported the fireworks test to jackal, and by extension mintia2, as well

#

and it managed to catch a bug in the xr/emulator

flint idol Sep 16, 2024, 6:29 PM

#

ornate ginkgo lol everyone is porting nanoshell fireworks

I'm writing my own based off of boron fireworks

limber wave Sep 16, 2024, 6:53 PM

#

inland radish also fun fact, I ported the fireworks test to jackal, and by extension mintia2, ...

wasn't mintia2 in it's early development stage?

inland radish Sep 16, 2024, 6:55 PM

#

it is

#

technically it supports modules (you can load a system debugger) but I didn't know that at the time so I ended up making it part of the kernel

flint idol Sep 16, 2024, 10:56 PM

#

I now continue making fireworks

lean glen Sep 16, 2024, 11:02 PM

#

ornate ginkgo yeah, thats why I specified gprs

well if cr2 is like 0xfefefefefe and RAX is the same, you know the problem is on accessing rax

ornate ginkgo Sep 16, 2024, 11:08 PM

#

Still not helpful, what is rax in that case?

#

It doesn't tell you anything "page fault at <pc> with flags xyz" doesn't tell you

flint idol Sep 16, 2024, 11:08 PM

#

I mean you could use objdump to try and figure out how rax got that way

#

I've done that a couple times

flint idol Sep 16, 2024, 11:08 PM

#

flint idol I mean you could use objdump to try and figure out how rax got that way

rather

#

what rax is supposed to be

#

anyway another reason to print the gprs is to make panic scary

#

https://media.discordapp.net/attachments/1141057599584878645/1141778523552493699/new_kpanic.png?ex=66e9b59d&is=66e8641d&hm=fa8883bbc4da6ad83f949e63c5ad3e084e6fa6cf418e8467ae5bfa7fd95cf33f&=&format=webp&quality=lossless&width=667&height=500

#

just found a really old screenshot

#

of obos

lean glen Sep 16, 2024, 11:10 PM

#

ornate ginkgo It doesn't tell you anything "page fault at <pc> with flags xyz" doesn't tell yo...

true

flint idol Sep 16, 2024, 11:14 PM

#

damn this test might kill the kernel

#

// Explode it!
// This spawns many, many threads! Cause why not, right?!
int PartCount = Rand() % 100 + 100```

flint idol Sep 16, 2024, 11:40 PM

#

aaand my timer interface has decided to die

#

so uhh

#

the timer irq

#

has decided to randomly to get masked

#

how?

#

idk

#

why?

#

idk

#

because of my damn irq interface

#

fixed

#

I guess

#

vale nymph Sep 16, 2024, 11:58 PM

#

thats a beautiful #when-your-os-goes-crazy moment

#OBOS (not vibecoded)

Abandon all hope, ye who enter here