Aster

[final sable]

Aster is my kernel, which I have been working on for the past ~2 months. It's probably my 3rd attempt at writing an OS, the first two being 2-3 years ago. I thought to revisit osdev after I have learned more stuff about computer science in the last years, mostly because low-level stuff never fails to spark my interest. I'm in my highschool final year so time is limited, as I'm also working as a dev in my free time. Here's what I have implemented so far:

• Kernel logging interface with log levels
• Flanterm, Serial, QEMU Debugcon
• GDT, IDT shared between all CPUs
• PIC disabling
• Interrupt handlers
• PMM, based on a bitmap allocator
• VMM, really simple, just maps and unmaps pages or ranges
• Kernel heap based on a first-fit allocator, for general purpose allocations
• ACPI tables parsing (might port uACPI though)
• MADT parsing
• Timer interface which uses the HPET timer if available, otherwise it fallbacks to the PIT
• LAPIC initialization with support for xAPIC and x2APIC & NMI routing as per MADT
• LAPIC timer calibration with HPET or PIT, support for one-shot and periodic modes (no TSC-deadline for now)
• IOAPIC initialization with all external interrupts handling done by the BSP (no IRQ balancing for now)
• APs initialization and per-CPU data
• IPIs
  The objective is to be able to run bash, gcc and most software which is intended to run on UNIX-likes.
  The name „Aster” comes from the name of the aster flower.
  Source code at https://github.com/Aster-OS/aster. Unfortunately there isn't a README yet.

[fathom orchid]

https://tenor.com/view/senator-palpatine-anakin-skywalker-phantom-menace-gif-10607636

[delicate tulip]

indeed! good luck

[wraith umbra]

good luck

surprisingly close to astral

its always funny to see that people actually calculate these acpi table checksums when even NT doesnt care about them

[brazen plume]

GDT init IDT init <3

[wind otter]

neat :3

[final sable]

ty ty

[final sable]

ty mint

[final sable]

is this about the name itself or the code

I chose the name by opening a greek dictionary to a random page lol

aster in greek means star and I thought it's cool

then I searched on google and found there's a flower (which looks pretty nice) with this name

so I said to go with this name

a few days later I find out about astral

[final sable]

that makes it even more related to astral lol

if you're talking about code, I didn't understand astral's code so there's no way they're that close

jk I did

[final sable]

if NT works then it works
it's probably done on behalf of the wiki's advice, I don't recall

[wind otter]

oop

[final sable]

@wraith umbra sorry for double-pinging you but I think you might've missed that question

[wraith umbra]

name

[final sable]

aight

[final sable]

so before I write my scheduler I want to abstract all devices I use for my kernel logs
that'd be flanterm, serial & debugcon (debugcon is not pushed yet)
currently at the start of kmain there is a klog_init(flanterm_log_lvl, serial_log_lvl) call which initializes flanterm & serial and sets the respective klog levels
I think I'll wrap all these devices in a struct and maybe play with the kernel cmdline so they can be enabled or disabled, I'll see

also, is something like tty a good name for this abstraction?

I think yes

[final sable]

alright got a way of abstracting TTYs

it's just the log levels missing 😛

the cpu brand string idea is stolen from obos

on line 2

so ty @mystic sun lol

[mystic sun]

Yeah I figured it would be useful in panics too

For bug reports

[final sable]

makes sense

[final sable]

it's slow though

it's like

20k logs per sec

[final sable]

I have some ideas on how to improve performance on it

also maybe eyalroz/printf would be faster than mpaland's

there's also a snprintf call to print the log level prefix which I could replace with a memcpy

[mellow hare]

Theres also nanoprintf

Don't know about its speed though

[final sable]

I'll look it up, ty

[final sable]

unless flanterm+printf is that slow, honestly I haven't done tests before I started working on this

maybe I should

[mellow hare]

And potentially Frigg's formatting thing, although not sure if that's in the printf style

[final sable]

so they have a different printf-like thingy?

[mellow hare]

Always worth testing when it's about performance

[mellow hare]

I remember seeing something about it, but I couldn't tell you more right now

[final sable]

it's fine

[delicate tulip]

i used to use eyalroz printf but i have been converted to nanoprintf too

it's just less fat, not sure about performance but i wouldn't be surprised if it was quicker, not that performance should be a blocker though

[final sable]

I have just taken a look at nanoprintf

and it seems pretty nice

maybe I'll add it to aster

[final sable]

@delicate tulip have you ever benchmarked flanterm to see how fast it is with reasonable use? in logs/second

I get like 30k logs/sec

[delicate tulip]

it's pretty fast

[final sable]

okay

[acoustic hatch]

Is the name of this project a reference to Halo (Project ASTER)?

[final sable]

I might have no idea what that is...
googling it shows I truly have no idea what it is

so, in short, no 😄

[final sable]

.

[final sable]

flanterm really is fast

I have finished abstracting TTYs and if I print to flanterm, serial & debugcon I get 1.5k logs/second

I/O is reaaally slow

with only flanterm (+printf) enabled it's 25k/s which is close enough to the 30k/s I got when I was calling flanterm with no printf to test how fast it should be

why do I care about these logs/second? idk

hopefully tomorrow I'll port nanoprintf cause it seems nice

then I'll start working on my scheduler

[final sable]

but before that I need to read about processes and threads

probably OSTEP Virtualization

right now I'm not sure what I'd put in my proc and thread struct

I have an idea but the biggest problem is I don't know what functions should my scheduler provide, aside from the context switch itself

at least I know I want to implement a RR scheduler

and when choosing between per-cpu queues or a global queue, I thought it's simpler to start with a per-CPU queue so threads don't switch CPUs and I don't have to lock the global queue

[mellow hare]

great starting point

and the per-core queues with no work stealing/sharing/migrating is a goood idea too imo

[final sable]

#1230349543623757845 message
note to self on why saving regs on the stack is better than storing them in the thread struct

[final sable]

warning: unused variable 'unused'

sometimes we do a little trolling

[final sable]

switched to nanoprintf and logs are still as fast as they used to be

with mpaland printf

after I push this one I'll start researching about schedulers

but I have a kind of national exams next week so I'll see what I can do

[final sable]

had some work to do lately but I've upstreamed the changes which replace mpaland/printf with nanoprintf

typedef void (*npf_putc)(int c, void *ctx);

does anyone have any idea why npf's putchar takes an int instead of a char

[wraith umbra]

because thats what putc takes in

[final sable]

oh wait there's also EOF

alright it makes sense

[wild bloom]

cuz putc is taking int afaik, but int because unicode shit

[final sable]

something's been on my mind recently
https://github.com/Aster-OS/aster/blob/4b604e827ba0a668c1e275162fe80ed694a3aa21/kernel/src/acpi/hpet.c#L84
here you can see that I do HPET sleep by polling the main counter value, however
if the main counter is 64-bit then that's fine probably, but if it's only 32-bit capable then it would wrap around every ~42 seconds. if sleep is called right before the main counter wraps around, hpet_counter_target could be > UINT32_MAX and it will spin forever I think
so... do I have to use timer comparators? I was lazy and I thought I'll be able to avoid them

I initially thought all modern systems have a 64-bit main counter but it turns out my PC's HPET does not support it

I'm so done with ACPI

even if I choose to use timer comparators they seem to be a pain

When the interrupt is generated, however, the hardware will increase the value in comparator register by the last value written to it!

[wraith umbra]

all amd systems have 32-bit hpet counters i think

[final sable]

I do have an AMD CPU so if it's true that makes sense

[mellow hare]

You can bypass that mechanic and write directly to the increment register (I forget what it's called)

So you can set the expiry time and the amount it should increment the comparator by in periodic mode

I wouldn't use the hpet for interrupts though, use it for calibration of other timers and gtfo

[mellow hare]

You can detect the wrap around, and for this purpose you shouldn't need it for more than 42 seconds

[final sable]

I just woke up and this was my first thought so I'll implement this and be done with it

[final sable]

pushed support for 32-bit HPETs

[final sable]

progress will be paused until I finish reading OSTEP virtualization & concurrency and maybe some other resources
this book is great

[final sable]

not related to Aster itself but rather to my journey on reading OSTEP
wrote a program to time a context switch and this feels so fun
avg ctx switch took 10.475000 us

given I'm doing 4 syscalls per loop and I've determined that a syscall takes ~1us, that leaves 10 - 4 = 6us as the time it takes to do a context switch nvm I forgot to divide by two

which is close to what a google search reveals it should take, i.e. around 5 microseconds

boi these things are fast

avg ctx switch took -1.389100 us nah I need to rdtsc

instead of gettimeofday

[brazen plume]

Context switch so fast you went back in time

[final sable]

avg ctx switch time: 5.487500 us [using gettimeofday()]
avg ctx switch time: 5.190810 us [using TSC]

except for the rather rare instances where gettimeofday delta time returns some negative value, it's pretty close to the one computed with the TSC

I got too caught up with this and forgot to actually read the next chapters

[final sable]

https://gist.github.com/el-zorab/b2855696e3143f4d77dcbf332ff40007 if anyone's curious

pass the number of context switches (min 2) as the first arg

conclusion: the cost is around 4-5us

(on my system)

[light zephyr]

you can emulate 64-bit HPET if only a 32-bit HPET is available. there's an example of this in my kernel at arch/x86/kernel/hpet.c iirc

basically it depends on hpet_read_counter being called more often than once per 20s (roughly half of wraparound), which I'm sure you can achieve (or move on to a better timer like the TSC)

[final sable]

I'll soon look into that, ty!

I remember that someone here briefly described the MLFQ algorithm but without mentioning anything about MLFQ, or maybe I had partially read that message, so I didn't know it's not their idea
I was so amazed at how great the workings of this scheduling algorithm were, only to now find out that it's actually pretty old and it has been around for a while 🙂

[final sable]

TODO for tomorrow: check (assert) that Limine responses are != NULL

college admission is 5 months away so I'm running low on (free) time

[final sable]

Positive nice values imply lower priority and negative values imply higher priority; when you’re too nice, you just don’t get as much (scheduling) attention, alas.

[final sable]

I've caught a flu sadly

and I also have some exams in the next 2 days, part of Romanian Baccalaureate

which are more of a joke that anything

one might say the same about Romania in general

however hopefully I'll finish OSTEP Virtualization this week

[final sable]

note: a few documents to read (one day) on memory allocators (from OSTEP references)

glibc malloc explained https://sploitfun.wordpress.com/2015/02/10/understanding-glibc-malloc/
The Slab Allocator: An Object-Caching Kernel Memory Allocator https://people.eecs.berkeley.edu/~kubitron/courses/cs194-24-S14/hand-outs/bonwick_slab.pdf
Hoard: A Scalable Memory Allocator for Multithreaded Applications https://www.cs.utexas.edu/~mckinley/papers/asplos-2000.pdf
A Scalable Concurrent malloc(3) Implementation for FreeBSD https://papers.freebsd.org/2006/bsdcan/evans-jemalloc.files/evans-jemalloc-paper.pdf
A Fast Storage Allocator” by Kenneth C. Knowlton https://dl.acm.org/doi/pdf/10.1145/365628.365655
Dynamic Storage Allocation: A Survey and Critical Review https://users.cs.northwestern.edu/~pdinda/ics-s05/doc/dsa.pdf

extra:
A Survey on the Interaction Between Caching, Translation and Protection https://cgi.cse.unsw.edu.au/~reports/papers/0321.pdf

[mellow hare]

Nice stash, if you want more reading material the on topic channels usually have pins of PDFs people have found (virtual memory and filesystems in particular)

[final sable]

seems like I was just using gettimeofday wrong

no one dared to look at my code to point it out

does that happen most of the time you don't read the manual with patience? probably

now I'm timing TLB hit and miss time

soon I'll be done with virtualization

and soon™ I'll be dreaming all calculus if I solve one more problem

[final sable]

least cursed graph

what have my hands done

1 6.660000
2 6.260000
4 8.780000
8 10.775000
16 12.148750
32 7.021250
64 8.714375
128 8.117812
256 7.776328
512 7.905234
1024 11.337090
2048 21.521230
4096 21.809473

ostep said
Visualization usually makes the data much easier to digest; why do you think that is?

but it looks so bad

[final sable]

tried to drive my car today but injector 1 said no

fun 300 metres

[mellow hare]

rip, at least that can be a cheap fix if its just the injector?

was it a VW?

[final sable]

yes it is

[final sable]

fortunately it's not the injector itself but rather some wiring which short circuits and that injector practically turns off in that split of a moment

causing pretty powerful vibrations in the whole car

especially when accelerating quicker than usual, the engine vibrates harder which in turn makes the cables move which short circuits and then you it gets jerky

so it's kind of a chain reaction

anyways next week i'll get it fixed

[mellow hare]

lucky it wasnt adding extra fuel, that could be so much worse.

yeah nice, well glad you're back on the road soon

more time for osdev in the meantime 😛

[final sable]

if it were for the injector itself it would have been a little bit costly to fix
the engine on this car is not a common rail one (meaning a high pressure pump sends pressurized fuel to the injectors which only need to spray it inside the chamber), but one with unit injectors which receive fuel only from the low pressure pump and pressurize it themselves before spraying it inside the cylinder
they combine the roles of the injection (high pres.) pump and of the common rail injector in one unit

so of course they are more expensive

[mellow hare]

Mate who designs that

[final sable]

oh wait it seems like it is the other way round

they are cheaper?

lemme look it up in detail

cause it seems counter intuitive

[mellow hare]

It makes sense to me that the injectors would be more expensive, theres more components to them.

[final sable]

yeah seems like that is the case

[final sable]

false alert lol

both have their problems though
common rail (CR) - more silent, better fuel efficiency, less pollution, injectors are less expensive since the high pressure pump is separate by design
however VW had some issues on CR engines where the high pressure pump would send little metal pieces to the injectors and damage all of them, they ended up replacing a lot of injectors for free because it was their fault
also since CR engines have a common rail with high pressurized fuel, if one injector is faulty and sprays too much fuel the other injectors might get too little pressure and in the end the engine might not start, it's just a disadvantage of the design
these injectors are cheaper but if the high pressure pump fails you, that one is really really pricy

engines with unit injectors are more pollutant, however these proved to be more reliable; in turn, unit injectors are more expensive than a CR injector
if one injector sprays too much diesel, the engine will probably still run so you can drive it to a mechanic

I dont think engines with unit injectors are made anymore

because of pollution regulations

[final sable]

no school for one week so that's a plus too

[mellow hare]

mm interesting, I'd never heard of unit injectors before, but I also know nothing about diesel engines (its all petrol based over here, except for trucks - but I dont own a truck to play around with lol).

[final sable]

once I heard around 75% of the cars in Romania have diesel engines

and indeed most people I know drive diesel cars

where do you live? I didn't imagine there would be a place without diesel cars

in turn I don't know much about the specifics of petrol engines

[mellow hare]

I'm in Australia

AFAIK diesel cars are a european thing

We do have some over here, but they're rare

[final sable]

I understand

[final sable]

could be

[final sable]

finished ostep virtualization 🥳

I need to read these tomorrow
Virtual Memory Management in the VAX/VMS Operating System https://users.soe.ucsc.edu/~sbrandt/221/Papers/Memory/levy-computer82.pdf
Meltdown and Spectre https://spectreattack.com/

I know of meltdown and spectre but I haven't read the original papers which is pretty sad

so I figured I'll take time to

[final sable]

I almost feel sad, it was such a great read

[mellow hare]

lol, you can only read it for the first time once 😔

[final sable]

yeah true

[final sable]

OSTEP concurrency time

[final sable]

I'm trying to switch to vim coming from a vscode background

and it's not easy (as expected)

however I am amazed at how powerful it is, for now I have to think of the fastest way to do something before I can actually do it, but with time it will get in my reflexes

I'd say I know some basic shortcuts for manipulating a single file

I aimed to only use Vim for the OSTEP homeworks and it was done with great success

what I've always found hard about Vim is working on multiple files

not because it really is hard, I just didn't spend much time learning how to

what I've used at one point when I tried to switch to Vim a few years ago was tabe and tabp to navigate between files

however I've done a fast google search and there seem to be better ways to

kernels/OS'es have "a lot" (subjective) of files so by working on my kernel I might also learn to efficiently switch between files

another problem which I'm sure will soon arise when I actually start working again on my kernel in Vim is the lack of auto completion, like I might forget what was the exact name of a function in some header or some variable

surely there exist plugins or idk what for this though

as I have read, Vim requires pretty extensive configuration and tweaking

and because of Vim's steep learning curve, I don't think Vim can be taught be someone else but instead one has to learn it by its own slowly and surely

but if anyone has some advice I could benefit from, I'd highly appreciate

[brazen plume]

not really

I use it as it comes from the distro because I cba to tweak it

[final sable]

so just the default everything? even the default theme?

at least now I know it's doable

[brazen plume]

yeah

[final sable]

hardcore

[final sable]

man SSD's became so cheap

I remember that when I've bought my first 120GB SSD it was like 0.8 EUR/GB

1 SSD GB was ~= 1 GB RAM

and now a 1TB SSD is 50 euros

[wind otter]

my vimrc is pretty small, i don't really need all those plugins and packages and stuff

[mellow hare]

another vim user

wait until you start trying to use motions and macros in other programs

[mellow hare]

thats where the fun begins, you can look into LSP (language server protocol). You can use mason (or a number of other plugins) to manage your installed/available language servers.

[mellow hare]

look into how to use buffers. :bp and :bn will switch to the prev/next buffer. You can map these to keybinds so you dont have to enter command mode every time (I personally use alt+</alt+>. Windows will help you manage your screen real estate.

also if you're using neovim rather than vim, I can recommend telescope for its 'pickers' (ways to navigate), in particular the live grep and file pickers.

[mellow hare]

look into marks: global marks are good for jumping around between multiple files.

[light zephyr]

this.

like I'll be in school writing a google document or something, and I want to jump up a few lines. instinctively I want to hit <esc>42k

[mellow hare]

or you'll be finding a few instances of :wa scattered around an email 😅

[final sable]

I have just taken a look at telescope and man, it's a game changer

feels even better than vs code

I truly wonder why I didn't embrace vim earlier lol

[final sable]

as I'm a few days far from finishing ostep concurrency I've already started imagining some of the API and internals of the scheduler

one thing I'm unsure about is what I would use a process list for (if anything)

because it seems like the scheduler mostly works on threads rather than processes, since threads are the smallest unit of execution if I can call them like that

and the only thing I imagine I would need would be a PID generator source, which can just be some variable I increment on every new proc

so, do I need to keep all processes on a list?

(or is it just the threads list that needs to exist)

thinking about Linux, there must be a process list because you can list them with ps

[mellow hare]

while your scheduler deals in threads, processes are useful for grouping threads and shared resources together. A process could be viewed as a live instance of a program. So you may not care about them for scheduling purposes, but you will for program management.

[wind otter]

it's so bad 😭

[final sable]

alright, so I may not need it right now but in future I will, ty

[final sable]

woah new color for introduced role

guess I'll have to introduce myself now

[final sable]

done

I'm also into photography and videography but I forgot to mention that lol

sometimes I feel like I enjoy too many things, but at least I don't get bored

[final sable]

I was searching for sth else but I came across this on SO

[final sable]

maybe I'll try booting Aster with Hyper after I do my scheduler

seems like it'd be fun to

[final sable]

great book which covers classical synchronization problems and how to use semaphores to solve them https://greenteapress.com/semaphores/LittleBookOfSemaphores.pdf

ofc the "great" is subjective

[final sable]

oh man

what a great journey on ostep concurrency

now it's time to actually write a scheduler

I'll try to write some initial things and then I'll probably take a break to actually make a list of the public and private API and global state I need to track

but not before installing neovim

[final sable]

pro tip: if you're having issues with pip, disable IPv6

why? yeah

if only I had listened to that SO answer way earlier...

[final sable]

I was getting the interrupt stack frame with the thread kstack mixed up but I think now all is clear

essentially I should make my ctx switch completely independent of any interrupt handling

so to make sure I don't mess up I'll write a cooperative scheduler where I'll call yield periodically in every thread

then add the timer interrupts part to make RR preemption

[final sable]

not so daily update: I think I need some VMA for allocating thread stacks

could also do something like allocate N pages and use the HHDM but nahhh

I could just use kmalloc?

just make it expandable (and shrinkable? altough that seems harder) cause rn it's just a fixed size (1MiB iirc) region at 0xffffffffd0000000

and it might run OOM

so either have some thread that expands it or do it on kmalloc call if it can't alloc or sth

second makes more sense probably

also someone suggested to use slabs but I'm lazy for that; actually not lazy per se but I put too much thought and heart into this freelist allocator and now I can't let it go

[final sable]

dude i have borked my ubuntu system so bad

I have deleted the known_hosts file

cause I wanted to remove some things from it and I said I'll just delete the whole file

(genius moment) - genuinely I don't why it messed up the system so bad

now after login it just shows a black screen

I have succesfully created a new user

here comes the best part

I can login on my main user only if I run sudo tail -f /var/log/kern.log on my new user in a TTY terminal

seriously, it doesn't work with anything else

not any other command with sudo or not

how in the world

[final sable]

disabling cups solves it?

I saw this in kern.log

[   14.935901] cups-proxyd[1865]: segfault at 18 ip 00005b304fec4d75 sp 00007ffd97fe9390 error 4 in cups-proxyd[5b304fec1000+7000] likely on CPU 0 (core 0, socket 0)

after ```
sudo systemctl stop cups
sudo systemctl disable cups

i can login fine

also I'm on low on disk space because of Astral's tens of GBs worth of space

[final sable]

fixed; my head hurts

reinstalled cups

[mystic sun]

lol why would it use that much disk space

[final sable]

ask the author

it's like 50gb

for a build that didn't even finish cause it was taking too long to

[final sable]

today I've worked on the scheduler

there's a lot of TODOs but soon enough I should have context switches done

[wind otter]

it's got a lot of ports and keeps the sources, afaik

[long rock]

galileo in the early 1600s

[final sable]

[final sable]

; void thread_switch(void *old_sp, void *new_sp);
.global thread_switch
thread_switch:
    ; push callee-saved regs to old stack
    push %rbx
    push %rbp
    push %r12
    push %r13
    push %r14
    push %r15

    mov %rsp, (%rdi) ; *old_sp = %rsp
    mov (%rsi), %rsp ; %rsp = *new_sp

    ; pop callee-saved regs from new stack 
    pop %r15
    pop %r14
    pop %r13
    pop %r12
    pop %rbp
    pop %rbx

    ret

maybe this would work?

woops asm comments start with # not with ;

forgor

[long rock]

yes it works and that's the way to do it (assuming you got the syntax right which idk cuz I don't do x86)

[final sable]

nice, ty

[final sable]

fixed 10ish page faults, another 10 to come

this fault's so bad even kpanic can't print info

time to fire up GDB

well it actually enters the thread's entry point

it just faults inside it on get_cpu()

oh get_cpu()->curr_thread is NULL cause I didn't set it

woops

[INFO] Creating thread TID 0 | bp ffffffffd0008348 | sp ffffffffd0008310
[INFO] Creating thread TID 1 | bp ffffffffd0010380 | sp ffffffffd0010348
[INFO] Switching from TID -1 to TID 1
[INFO] Thread exited. Halting indefinitely

TID -1 means old thread was NULL (since this is the first thread to run on the system)

anyways it's beautiful to have something working

working context switches for kernel threads 😄

rn it's cooperative (so it calls yield()) but tomorrow I hope to add timer interrupts to it

[final sable]

to this I added a check to see if old_sp is NULL and if it is, skip saving current thread state

to switch easily to the first thread, since the CPU's curr_thread is NULL initially

so it doesn't have a kstack to save to

for some reason stacktraces don't really work in kernel threads

well I don't do the function prologue and epilogue so that's probably why?

[final sable]

one advantage of per-cpu run queues with no thread migration is that there is much less data that needs to be thread-safe and locked

[brazen plume]

GDT Initialized <3

[fathom orchid]

I would put those in DEBUG too

[final sable]

what's up with it lol

[final sable]

this is actually the second time you mention it

[final sable]

they're gonna end up in debug anyways one day

for now, as oberrow (iirc) said, I want to see my thing alive

[fathom orchid]

nothing, some beginners just have IDT init ok and GDT init ok and it's a bit of a meme

I have gdt/idt init too

[final sable]

ah makes sense

[wraith umbra]

Initializing descriptor 1........ OK
Initializing descriptor 3... SUCCESS!!!!
Loading GDT.. OK!

[final sable]

you forgot the triple fault

[brazen plume]

we need a gdtos that completely ignores page tables, identity maps all of memory and only uses segmentation like god intended

[final sable]

inb4

[wraith umbra]

we need an os with most advanced gdt and idt init possible

[final sable]

I didn't do anything today cause I was busy with my bday
there's also a lot of things to do for school so it is what it is, I might spend these days getting them sorted out so I can focus working on this afterwards

[kindred spruce]

i wonder if xenix for the 286 did that

[final sable]

ofc I don't actually setup the stack correctly

stacktraces don't work because it PFs inside do_stacktrace()

but it has to acquire kpanic lock and it can't so it deadlocks

imma fix this first

[final sable]

which is inside kpanic

[final sable]

nvm I was actually not pushing 0 as a return address before anything

which my stacktrace relies on

because Limine does it

but this is not Limine's stack

anyways it's fixed

also I had a bug in my stacktrace function which made it even worse to debug but still I figured it out quickly (sort of)

[final sable]

I took a break to focus on some school stuff, I'll be back in a few days
while trying to fall asleep my brain clicked and I found a bug inside an asm thunk lol, gotta fix it soon

tldr I forgot rdx is not preserved across function calls

[final sable]

I made this to pass an argument to a kernel thread and then get the returned value

    // thread stack initial state
    *(--kstack_ptr) = 0;

    *(--kstack_ptr) = (uint64_t) arg;
    *(--kstack_ptr) = (uint64_t) start;

    *(--kstack_ptr) = (uint64_t) thread_entry;
    *(--kstack_ptr) = 0; // rbx
    *(--kstack_ptr) = kstack_bottom; // rbp
    *(--kstack_ptr) = 0; // r12
    *(--kstack_ptr) = 0; // r13
    *(--kstack_ptr) = 0; // r14
    *(--kstack_ptr) = 0; // r15

and an assembly stub

.extern thread_exit

.global thread_entry
thread_entry:
    pop %rcx // start
    pop %rdi // arg

    call *%rcx

    mov %rax, %rdi
    call thread_exit

    ret

and exit gracefully

I'm pretty happy with it

now I think it's time to schedule threads on multiple CPUs

man I wish I wasn't lazy and I actually parsed symtab to print the function names on kernel panic...

one day..

[final sable]

turns out I've introduced a bug in my kmalloc which went unnoticed because I never call kfree in my kernel

here, it should have been kassert(!get_flag(to_free, FLAG_IS_FREE)); instead; notice the !

this one was cheap

[fathom orchid]

needs more asserts

[final sable]

it's time to do this once and for all

I started reading the ELF wikipedia page and it turns out it's not even as bad as I thought, but rather pretty simple and elegant

reminds me of this pinned message

[final sable]

assert(kassert exists)

[final sable]

oh man I could've just used a struct

this hurts

[final sable]

you gotta love this

[final sable]

symbols, finally

[fathom orchid]

I assume that's for stack trace and/or relocatable modules?

[final sable]

stacktrace is what I wanna do

[fathom orchid]

do you parse them at runtime or extract at compile time?

[final sable]

I'm tired of having to manually copy the addresses and see which function they point to in the ELF on every stacktrace

[final sable]

runtime

[fathom orchid]

what happens when the kernel is compiled in release mode?

(no debug symbols)

[final sable]

in that case I think I'd return "<unknown symbol>" or sth, idk

if symtab or strtab isn't present

[fathom orchid]

that's the advantage of extracting symbols before stripping the elf

[final sable]

so you don't have to compile with debug symbols?

wait what do you mean by stripping the elf

[fathom orchid]

removing the symbols, but now I realise incbin ing a file of symbols defeats the point of stripping

[final sable]

ah fair

[fathom orchid]

I guess it removes some unused data, so not completely pointless

[final sable]

tbh I didn't think much of what I'd do for this

[fathom orchid]

that sounds very bad out of context lmao

[final sable]

lol

[final sable]

if it's gonna be a release mode might as well just disable kpanic

nah I'm kidding

[fathom orchid]

I disable asserts in release mode

I have if-then-panic for essential stuff

asserts for sanity checks (like for functions called only from the kernel with known or predictable arguments)

[final sable]

makes sense, maybe I should've followed this approach

how do you get debug symbols in your kernel?

[fathom orchid]

I haven't done it yet lol
but I just parsed the file at runtime in my previous kernel

I'll get around to doing it once I need symbol addresses for modules

gdb stacktrace works good enough for now

[final sable]

how are they useful or related to modules? I haven't got that far to use them so I know pretty much nothing

also I have a weird symbol lol: 189: 0000000000000100 0 NOTYPE LOCAL DEFAULT ABS i

maybe it's from an asm macro for ISRs?

[fathom orchid]

well I used relocatable object files for modules
you need to parse them and assign correct addresses for references to kernel functions

for example a module might call a logging function which has an address that is not known at compile time

[final sable]

but that doesn't makes sense, does it? aren't macros in asm similar to C? I thought the preprocessor would do the job and they shouldn't be visible in the object file

[fathom orchid]

you either have limited amount of hardcoded function names (why) or parse the symbol table

[final sable]

that makes sense

[fathom orchid]

should be

you could just use the c preprocessor

hmmm but then loops aren't easy

[final sable]

weird

its address is 0x100 so I doubt it is from C

i.e. not in the higher half

[final sable]

parsing symbols is pretty much done, I just have to clean up the code tomorrow

[mellow hare]

Nice one

Symbols are a helpful addition

[mellow hare]

The name is interesting, and i suspect the value is being stored as the address there, rather than it being at address 0x100.

[final sable]

ty

[final sable]

oh yeah, I think you're right

https://github.com/Aster-OS/aster/blob/3056bce6c1638624d0af4559a3c6ed68dd84fa94/kernel/src/arch/x86_64/interrupts/isr.S#L135
this is probably it, since at the end of the loop i is decimal 256 = 0x100

why does it end up as a debug symbol though?

assuming that's where the symbol is from

[mellow hare]

aha

[final sable]

I'm at school rn so I can't test it but I'll try to change the number of repts and see if the value of the symbol changes; if it does, that's where it's coming from

[mellow hare]

from the gnu as manual:

Set the value of symbol to expression. This changes symbol's value and type to conform to expression.

[final sable]

TIL 🙂

[final sable]

there are so many things I'm simultaneously working on at this moment that I can't even keep track of them

so I'll write everything here when I'm ready to push and all the bugs are fixed lol

[final sable]

managed to touch this today, added timer interrupts to scheduling

I think I have to fix some (yet silent) bugs by adding some locks or disabling interrupts in some critical sections, but other than that it's fine

now I see that using interrupt priorities would be useful

I'll think about adding it or not

timer interrupts in scheduling finally clicked for me

I thought that if you call your ctx switch inside your interrupt handler things will go boom, but

you enter the interrupt handler
you save interrupt state
you enter the context switch
you save old thread state
you switch to the new thread's stack
you restore new thread state
you return from the context switch

but this last return doesn't go back to interrupt handler, it actually goes to the ip the new thread was executing when it yielded

instead the the return from interrupt handler happens when the old thread, the one which was interrupted, is again scheduled

it all makes sense now 😄

and I see how it's easy to get these two mixed up

it's late so gn

[mellow hare]

yeah, just make sure you finish any interrupt-related business before switching to another thread (I'm thinking of sending EOI and running any callbacks).

[final sable]

sending the EOI after switching has already bitten me

what do you mean by callbacks?

[mellow hare]

lol, we've all done it I'm sure 😛

[mellow hare]

any driver callbacks for the device that might have sent the interrupt

its also something you would realize quickly

[final sable]

ah I get it

rather off-topic, but here's a pacman clone I've made less than a year ago. it's more of a highschool project, which I should've made in the 12th grade but I coded it a year earlier to have more time to prepare for exams and uni admission this year. revisited it because it's time to submit it and I have to add the sdl2 libraries to the .zip
it's actually pretty enjoyable to play

this is 4x sped up

the ghosts textures are kinda lacking but it is what it is :p

it also reimplements the original behaviour of each individual ghost

learn more at https://pacman.holenet.info/

this was my primary reference, the ghosts are pretty unique and the behaviour is rather complex for something which I thought would be as simple as following pacman with some added randomness

instead they depend on the current mode (scatter, chase, etc.) and maybe on the other ghosts' positions, which make them able to catch pacman from all directions in some scenarios

[fathom orchid]

I choose to believe you played that real time

[final sable]

pretty sure I did, I just didn't want to brag about it

[final sable]

cross compiling it for windows was harder than I thought

[final sable]

imma do doubly linked lists for the thread queue cause I want to make thread deletion faster

I haven't really thought about deletion until now so now I wish I had implemented DLLs from the beginning but also it's not that bad, I'm just a little lazy

good thing I can copy the insertion and deletion from my kmalloc impl without silently introducing some painful-to-debug bugs in these

I thought about making some interface (insert, delete) for DLLs such that I could pass a pointer to any struct, without it caring about what it really is (that'd be a void ptr)

but idk if that would be a good design

you'd have to guarantee the offset of prev and next inside that struct, other than that I think it would work

whatever, prolly not gonna do it

[final sable]

for dead thread deletion I made a worker thread and a distinct queue for dead threads
when a thread exits it gets removed from the main queue and added to the dead threads queue
a worker runs periodically and checks if any dead thread exists
if at least one dead thread exists, it deletes it from the queue and frees all associated memory; it does this until the dead threads queue becomes empty, then it yields
otherwise, it yields

if I had priorities it would be even faster as this worker thread doesn't need to run that often, but it's still faster than checking and destroying dead threads on every ctx switch or something

by priorities I mean some kind of niceness as in linux

[mellow hare]

maybe take a look at this for some inspiration: https://github.com/freebsd/freebsd-src/blob/main/sys/sys/queue.h. Mind the license ofc.

[final sable]

I'm on my phone rn but I'll take a look when I get home, ty

seems like the kind of thing I'm looking for

[mellow hare]

Its quite handy, there's also 'tree.h' too

[final sable]

while (thread) {
    klog_info("Destroying TID %llu", thread->tid);
    thread_queue_delete(&cpu->dead_threads, thread);
    kfree(thread->kstack);
    kfree(thread);
    thread = thread->next;
}

first one to find the bug in this snippet wins

took me a while...

[long rock]

Use after free

[final sable]

indeed

I went to bed immediately after finding this use after free cause I was tired of debugging
the behaviour caused by this bug was pretty weird so it was hard to debug
today I plan to reproduce the bug again and see exactly why it behaved the way it did, hoping to catch future bugs like this faster and make more sense out of it

[final sable]

vector 0x0d ec 0x21bc

weird

if I let the scheduler timeslice be 1ms it GPF's after 10-15s of running around 50ish threads

increasing it to 10ms makes it not GPF...

it's hard to track this one down

and that error code seems really off

wiki: The General Protection Fault sets an error code, which is the segment selector index when the exception is segment related. Otherwise, 0.

lowering the timeslice even further (100ns) does trigger the fault earlier on

[final sable]

I might just be well out of my mind

I think I was booting my kernel with SMP on, however I wasn't properly initializing the scheduler for the APs and some bad stuff was happening - still have to figure out what was going wrong

[final sable]

kernel's cursed

that's the current state of things

but I won't give up

[final sable]

ofc context switching with interrupts on would be fine, right? (/s)

the issue is I don't really know how to disable them and enable them properly

[final sable]

yeah idk what to say about this but I think I've managed to do it

I have a void cpu_set_int_state(bool int_state) which atomically sets the interrupt state and returns the old state

so I figured I can just call it inside the ctx switch, before returning

# void sched_thread_switch(void **curr_sp, void **new_sp, bool prev_int_state);
sched_thread_switch:
    test %rdi, %rdi
    jz skip_thread_save

    push %rbx
    push %rbp
    push %r12
    push %r13
    push %r14
    push %r15
    mov %rsp, (%rdi)

skip_thread_save:
    mov (%rsi), %rsp
    pop %r15
    pop %r14
    pop %r13
    pop %r12
    pop %rbp
    pop %rbx

    mov %rdx, %rdi
    call cpu_set_int_state

    ret

void sched_yield(void) {
    bool prev_int_state = cpu_set_int_state(false);
    lapic_timer_stop();

    // pick next thread, change thread states, change current thread
    // and some other stuff omitted

    lapic_timer_one_shot(SCHED_TIMESLICE, sched_vec);
    sched_thread_switch(curr_sp, new_sp, prev_int_state);
}

I do this because I'm not sure if I should reenable interrupts after the context switch or not, instead I just set them to the previous state

[final sable]

iirc I saw something like this in astral and thought it would be nice to have so I added it some time ago

I've modified my thread entry stub to align the stack on a 16-byte boundary before calling the start routine, idk if it's needed or not

if someone could say for sure I'd appreciate it very much

[mellow hare]

If it's for some assembly you can use pushf/cli/popf

[final sable]

hmm you're right

I might actually do that

sometimes it's the simplest things that don't pop into my head

anyways the scheduler's fixed 🙂

now I need to add SMP to it

[final sable]

done - it was pretty straight forward given I wrote it with SMP in mind from the start

and it runs fine or a variety of tests

I'll push everything soon

100 seconds of my scheduler, i.e. pure perfection - or far from it

idk why the recording is so laggy

can't even run two shells, a browser and a screen recorder

maybe qemu was stealing a lot of cpu time lol

[final sable]

I left out processes out now that I think of it

oof

should be a quick addition

add another linked list to have a list of threads for each process, apart from the per-cpu run linked list

switch cr3 if needed

otherwise don't, I think even if you reload it with the same value it flushes all TLBs so big no no

incremental PIDs and that's it for now

then I'll get to writing a VFS - or taking a break to actually focus on school for these last 3 months that I have left

we'll see..

[final sable]

oh and a global linked list of proc's ofc

linked lists ftw

[long rock]

No

To flush remote TLBs you need to u oee

[final sable]

to what?

[long rock]

You need o tlb shootsown

[final sable]

I was refering to all local TLB entries - nothing to do with the other CPUs

maybe I should've worded it better

like would loading a new value into CR3 invalidate all local TLB entries even if the new CR3 value is the same as the old one?

why am I asking this when I can read the sdm

[mellow hare]

self-rtfm

[final sable]

introducing some macros for linked list node insertion and deletion rn

cause I got tired of rewriting the same code for the 4th or so time

also it doesn't help with readability so it's a nice addition

also working on adding processes

[final sable]

the national baccalaureate exam simulations take place these days so, again, I'm a little busy

[final sable]

this was today's exam if anyone's curios - you've got to translate the questions though

I was so close to actually getting a full mark on it

I've managed to mess up the answer on this
Determine the probability that, choosing a number n from the set of two-digit natural numbers, n^2 will be a three-digit natural number.

which was honestly the (second) easiest question on the paper

sometimes we overlook simple things

[final sable]

I can't keep track of the number of times I've fallen for this

my_function() {
  spinlock_acquire(...);
  if (*certain side-case cond*) {
    // handle differently
    return;
  }

  // handle general case
  spinlock_release(...);
}

i.e. deadlock caused by returning early and forgetting to release spinlock

OSTEP provided some studies on deadlocks and most of the bugs were caused by stuff like this (iirc they considered some well known software, like DBMS, Linux, Mozilla, etc)

and when I read it I said to myself that I'll remember this and never fall for it

[final sable]

^ indeed

needless to say, I got humbled instead

added processes to the scheduler

vfs time?

I need to push changes first but that's for tomorrow

[final sable]

changed some things here and there, now getting debug exceptions

funny how I find the cause the second I mention the issue I'm having

[final sable]

unfortunately I have to put this project aside for the next months, in order to prepare for university
I'll be back in mid july, after taking the entrance exam and hopefully getting accepted 🙂
I've pushed all changes, scheduler has all the features I wanted to add for now, only thing it's lacking are user threads but I think it's logical to leave them for later, they should be easy to add with the way I've organized everything
here's the latest commit https://github.com/Aster-OS/aster/commit/90ad784ab9fd8828d9b39c8c9d50fcb2712a9d42

can't afford to start reading about filesystems now cause I know how addicting coding acquiring knowledge is for me (and time's extremely precious rn)

until later, good luck to everybody

[mellow hare]

Good luck with uni!

[void peak]

Lmao I did that the other day in my slab allocator..... I understand entirely how you feel

[final sable]

thank you!

[final sable]

it happens to all of us, in the end everyone just keeps becoming better and better by making mistakes

[final sable]

in mid july I was accepted at the faculty I want to study at
right now I'm tinkering with a lot of stuff, I've bought an oscilloscope, I'm fixing an analog audio mixer, building a variable lab supply and a function generator, and some other things, so I'm kinda caught up, but I'll start reading the Persistence chapter from OSTEP whenever I get the chance 🙂

I'm in a spot where many things I've wanted to build/do have acummulated since I was focused on preparing for the entrance exam and I didn't have too much free time, and now I have to choose from multiple things which I equally like
it's not an easy choice, but I'm sure osdev will be among the things that I'll keep doing

I feel like I forgot some parts of my so barebones kernel so before anything, I should read the source code and refresh this stuff in my brain

[mellow hare]

nice and congrats!