Ultra

[wide ether]

ultimately it's mostly a gimmick

[ornate torrent]

this is it's biggest disadvantage

because with it, I can be at foo initialization stage and not know if bar has completed yet.

[wide ether]

huh

[ornate torrent]

this isn't a problem per se

[wide ether]

i'm pretty sure it can

[ornate torrent]

but little assumptions sneak in here and there

[wide ether]

that's like the entire point, no?

[ornate torrent]

yeah but you have to add an edge to your dependency graph for that

[wide ether]

which you're supposed to

if you only care about being able to make heap allocations, you can depend on the memory stage

[ornate torrent]

giving you the choice of adding only the edges you need also gives you the choice of adding too few.

[wide ether]

sure, but so does putting a call at the wrong line

[ornate torrent]

yup, but that has a higher probability of getting caught early on

[wide ether]

look at this beauty

[ornate torrent]

if I did DAG based initialization, I would basically still have everything depend on all the things I put before it in sequential initialization

[ornate torrent]

oops, panic in system.acpi but generic.fbcon didn't run yet so you get no output

[wide ether]

well tbf explicit initalization is even better than what i had in v0

[ornate torrent]

also doing scheduler setup before you know what the SMP topology is can be tricky

[wide ether]

scheduler setup here only means creating a kernel process

[ornate torrent]

and the VFS probably wants to tell the time for atime ctime mte etc.

it's easy to implicitly assume that something's there because in any realistic sequential initialization you would put it really early but the depgraph doesn't guarantee it

[wide ether]

i guess i won't be carrying this over to C

[ornate torrent]

well it might be considered a design decision

[wide ether]

i did it for fun

for the cool graph visualization

[ornate torrent]

with really strict layering it's probably more reasonable to do an initgraph

[ornate torrent]

bro just dump the function call graph into graphviz like a normal person

[wide ether]

well that's what i do

[ornate torrent]

or like in a microkernel environment

[wide ether]

[ornate torrent]

no not the initgraph

[wide ether]

wouldn't that just look like a string though 😭

[ornate torrent]

void bar();
void foo()
{
        bar();
}

creates an edge foo -> bar

[sterile kayak]

how come TSC doesn't depend on any calibration sources?

[wide ether]

invariant tsc

[sterile kayak]

still need to calibrate it

[ornate torrent]

you mean the intel CPUID meme?

[wide ether]

yes

[ornate torrent]

because invtsc is something else entirely

[wide ether]

that's why it fails 99% of the time

i only really use the hpet

(bad)

[ornate torrent]

and AMD doesn't have that information

[uneven abyss]

"We have menix progress report channel at home"

so are you gonna rewrite then

[ornate torrent]

yeah this discussion should probably be migrated over there

[livid talon]

^ Marvin recorded his goodbye appeal to rust. 😁 tools should help one to create, not destroy one from the inside.

[prime wraith]

@sterile kayak i really like your initgraph idea, did u invent it yourself?

[sterile kayak]

well the concept itself is from mammogram

but the design and implementation is mine yes

[prime wraith]

interesting

i was gonna do the normal initcalls stuff, but that seems hacky compared to this

[sterile kayak]

what is the "normal initcalls stuff" in this context

just explicitly calling stuff in the right order?

[prime wraith]

nah, same idea as yours just no dependencies, but hardcoded init levels

like linux does

[sterile kayak]

ah

[prime wraith]

and stuff within the same level cannot depend on each other

well they do sometimes because linux depends on the link order for some things iirc

which is insane

[sterile kayak]

I do have that as well with the early, init, earlyap, initap stuff

I just also have deps within each level (I call them targets iirc)

[prime wraith]

why do u need different levels?

cant that be solved with dependencies?

[sterile kayak]

They run in different contexts

early: runs in the kernel main function, which later becomes the boot cpu idle thread
init: runs in the kernel init thread
earlyap: same as early, but runs on each extra cpu instead of on the boot cpu
initap: same as init, but runs on each extra cpu instead of on the boot cpu

if you have a design where the kernel main function becomes the init thread you only need init and initap

[prime wraith]

ohh thats pretty smart

[prime wraith]

and you don't?

[sterile kayak]

nah

[prime wraith]

why not?

[sterile kayak]

not really a specific reason for it

it's just what I did ig

[prime wraith]

so u have every AP spawn a separate thread to run init tasks?

[sterile kayak]

the init thread migrates itself onto each ap where it runs initap before migrating itself back onto the boot cpu

[prime wraith]

ohhh lol

so what happens to it in the end

[sterile kayak]

at the end of smp init it's migrated back onto the boot cpu

at the end of init it becomes the kernel event loop (just a thread that executes random bits of kernel code that can't run in IRQ or DPC context but isn't worth spawning an exclusive thread for)

[prime wraith]

like a softirq thread?

[sterile kayak]

I guess? I'm not super familiar with the softirq stuff

[prime wraith]

iirc linux has those per-cpu

just general purpose work "threads"

[sterile kayak]

for example on alarm() timer expiry some mutexes need to be taken so the timer expiry function (which runs in IRQ context) schedules a task for the kernel event loop thread

[prime wraith]

interesting

what does alarm do exactly?

[sterile kayak]

on a kernel that doesn't need such a thread the init thread would probably eventually issue an execve

[sterile kayak]

send SIGALRM after x seconds

[prime wraith]

execve into what?

[prime wraith]

ah

[sterile kayak]

/sbin/init

[prime wraith]

ah

makes sense

[ornate jasper]

now yall are making me think about initgraph type stuff in zig

like the toposort to get the order could be done comptime probably

[prime wraith]

yeah im thinking about how that could work

also seamlessly with dynamic modules vs compiled in modules

[ornate jasper]

well you couldnt do comptime sorting dynamically by definition

and zig's features dont marshal across link boundaries at all so dynamic would be difficult

[prime wraith]

okay i was wrong about linux having per-cpu IDT

it has one IDT but then per-cpu dispatch table for normal IRQs

that makes sense actually

typedef struct irq_desc* vector_irq_t[NR_VECTORS]; wait u can typedef an array?

wtf

[twilit geode]

Interesting

[stuck agate]

oh yes

C declarations/declarators are fun

[prime wraith]

i feel like linux is really missing abstraction in a lot of places

like it has literal apic.c doing irq allocation calling into generic code

why not have generic code do the allocating and call into hw to map it

so much duplication there for no reason

i noticed this in their unwind code also

clearly arch independent stuff is duplicated per arch for no reason

[carmine token]

We also do that in Managarm

IRQ controllers are too different to make generic IRQ allocation possible

[prime wraith]

btw do u happen to know how this works on aarch64

[carmine token]

Remember that GSIs are an ACPI concept and don't exist on all systems

[prime wraith]

does it have anything like an idt

[carmine token]

No

It has two vectors

One for irqs, one for fiqs

[obtuse sparrow]

Yes

[prime wraith]

so like, is there a way there to know which device caused the irq without asking every single one?

[carmine token]

FIQs are higher priority irqs but not really used on general purpose systems

[carmine token]

You ask the irq controller

[prime wraith]

what does it give u?

[carmine token]

A number

Depends on the irq controller

[prime wraith]

so there's still some sort of a way to map irqs to a number?

[carmine token]

For the GIC, there are both core local and global IRQ numbers

But you obviously can't access the core local IRQs of other cores

[carmine token]

Some way? Probably yes, by doing your own translation

But there's no canonical way

And it's generally not useful, why would you need to do it?

[prime wraith]

uhh, so that u dont have to query every existing device on the system?

[carmine token]

You don't have to do that

You know which device is connected to which GIC (or other irq controller) input

[prime wraith]

so it's hardcoded?

[carmine token]

For simple systems, yes

MSIs are not hard coded

MSIs are a bit more complicated than on x86

But idk the details of MSI handling

[prime wraith]

i've looked at the gic code but its like tons of files and different versions

hard to make sense of it

[carmine token]

This also interacts with the SMMU if there's one

[prime wraith]

smmu?

[carmine token]

Arm's iommu

[prime wraith]

ah

[carmine token]

^ also one difference to x86 is that IPIs go through the GIC as well

[prime wraith]

how is this different to x86?

[carmine token]

My recommendation would be to represent an IRQ as a struct that knows how to handle it or that you can attach handlers to etc instead of a number

That is easier to deal with, for example in situations where irq controllers are nested

[prime wraith]

u just misunderstood what i said, i was never gonna represent it as a number

but IPIs on x86 go through the lapic anyway right

whats the difference

[carmine token]

On x86, IPIs go to the CPU directly (i.e. to the lapic and then to a specific idt vector)

[prime wraith]

and GIC is like IOAPIC?

[carmine token]

So you don't have to ask an irq controller to determine if an ipi happened

[prime wraith]

or is there like a per-cpu gic also

[carmine token]

The GIC has both core local and global interfaces

[prime wraith]

so what you're saying is IPIs are more like an external interrupt

[carmine token]

Yes

[prime wraith]

ah

[carmine token]

There's no difference between IPIs and external irqs except that the possible range for ipis is core local

[prime wraith]

I see

kind of annoying that this stuff is so much more complex than x86

well i guess its like the ISA bus where every irq is hardcoded

[carmine token]

Another difference to x86 is that GIC irq numbers are not contiguous

So that's also an assumption that should be avoided

[prime wraith]

thats probably why they dont use irq matrix thing on aarch64

its only used on x86 and looongarch

they basically have apic.c have the generic allocator stored inside

behind irq_domain->alloc

and gic just does whatever crap

[gentle sandal]

MSIs in aarch is honestly terrible

Or rather MSI with gic

[prime wraith]

why is that?

[gentle sandal]

Its an over engineered workflow imo, requires multiple data structures (5 iirc) in total and is extremely different then normal irqs

You can search LPI in the gicv3 spec

[prime wraith]

damn

[carmine token]

Iirc at least if you have the smmu on, you get one msi space per device

Or per stream ID or whatever

But I'm not that familiar with MSI on the GIC

[prime wraith]

bruh do they get paid for more added complexity

if u strip arm support from linux it would probably be simple af

more stuff could be generalized

there's like so many arch-specific helpers that only exist because of aarch64

[gentle sandal]

Do you have an example? Im not familiar with linux source

[prime wraith]

a big one i can think of is paging related stuff https://elixir.bootlin.com/linux/v6.15.6/source/arch/arm64/include/asm/pgtable.h

this stuff here has sooo much extra logic to compute contiguous pages for example

since aarch64 has an optimization for it

• aarch64 is the only reason there's a whole bunch of barrier or other hooks for this logic

there's like a bunch of functions u can click on and you will see two things: asm-generic/thing.h arm64/thing.h

sometimes also s390 and other weird things

had to look up what that even is

[gentle sandal]

Yeah that makes sense, the translation code has a lot more variations then other archs so it makes sense it's often unique like that

s390x is fine too, iirc the translation is kind of standard but it has some quirks with prefix

[prime wraith]

like they dropped itanium but this arch that was abandoned in 2004 is still fine somehow lol

[gentle sandal]

s390x is not abandoned

[prime wraith]

wait really

Where is it used?

[gentle sandal]

Ibm mainframes

[prime wraith]

as far as i can see the later ones use https://en.wikipedia.org/wiki/Z/Architecture

[gentle sandal]

I think it's essentially s390x with a different name

There's differences between s390 and s390x but i never cared to learn about them

[prime wraith]

ah ok

[gentle sandal]

Just to make sure, random commit from linux s390

[prime wraith]

yeah i guess its corpo backed still

[vestal zealot]

Fun fact: Syntactically, C lets you do shit like declare an array of functions.

Not pointers. Just functions.

This is not semantically valid of course but it's still syntactically.

[ornate jasper]

thought it was four

also on aarch64 the entries of the table are raw machine code, either sized for one instruction or 128 bytes depending if youre on M or A

on A ive usually seen the code be a bunch of register pushes followed by a call, on M afaik its usually an unconditional jump instruction to the actual vector

[carmine token]

There are more than 2 entries in the vector table but only two are for irqs

[ornate jasper]

fair distinction

[ornate jasper]

zig even lets that be semantically valid actually - raw functions only exist at comptime but you can put them in an array and the entries will just be how you refer to the function in code, and i think it even works if you do addressof the array entries to get function pointers. it might even be nice and generate the functions in a row in memory, it did for my use, but dont think its guaranteed

arrays of functions (not pointers) is still kinda dumb though

[vestal zealot]

Stealing this for the custom programming language I'll inevitably make in Lily-CC

[ornate jasper]

nice. yeah its nice if kinda dumb sometimes. in zig it only works because of comptime evaluation but it does work

i refactored it out of my code though, so im using an array of function pointers instead

i was using it for my IDT entry functions

realistically if you can get an interrupt number of some kind from the controller then having only one interrupt vector for IRQs (or two for aarch64 with the FIQs) is fine, given most of the time all your per-irq handler does is push the interrupt number and call a common function with that info added

[wet quail]

Is this written in C++?

[prime wraith]

hell no

[wet quail]

Ok

[prime wraith]

@sterile kayak why do u need a .org and .size here?

[sterile kayak]

I like having all my symbols have a correct size in the symbol table

[prime wraith]

oh is this like the size of the last defined symbol?

[sterile kayak]

And the .org is to ensure both that there are no thunks for vectors >= 256 and that the array is expanded to 256 entries if not enough thunks were declared

[sterile kayak]

.size a, b sets the size of symbol a to b

[prime wraith]

ohh

and .org?

for some reason i dont see them in the manual

[sterile kayak]

.org x sets the offset for the current section to x

and errors if the offset is above x

[prime wraith]

ohh

smart

[sterile kayak]

https://sourceware.org/binutils/docs/as/Org.html

https://sourceware.org/binutils/docs/as/Size.html

[prime wraith]

i was looking here https://sourceware.org/binutils/docs/as/AS-Index.html

it is there, but apparently without .

btw are u still working on it?

[sterile kayak]

recently the moments where I have motivation and the moments where I have time have not coincided unfortunately

but it's not abandoned or anything

[prime wraith]

Yeah thats fair

[prime wraith]

@sterile kayak any reason u do it like this?

why not just

push rax
.cfa_rel_offset rax, 0
.cfa_adjust_cfa_offset, 8
...

[sterile kayak]

this looks cleaner imo

[prime wraith]

but it's not atomic this way

if it dies in one of the pushes u will have no hints

[sterile kayak]

it is, because the default is same value

[prime wraith]

ah right

[sterile kayak]

no saved registers are overwritten until the rules are in effect

[prime wraith]

btw are movs faster than pushes?

[sterile kayak]

not sure

[prime wraith]

hm

[sterile kayak]

on modern CPUs it's probably about the same

[prime wraith]

i was thinking of doing this just because its easy to macrofy and u dont need to calculate offsets manually

[sterile kayak]

fair enough yeah

the sub-then-mov thing does make it easier to tell whether you're missing some fields in the struct def or in the assembly imo

because the final offset has to match the size of the struct

[prime wraith]

fair

[sterile kayak]

but it's all personal preference

I wouldn't expect there to be any actual performance diff between this and a bunch of pushes

[prime wraith]

yeah probably

[sterile kayak]

one thing that will be noticeably different is cfi size

with this you have one advance_loc and a bunch of offset ops

with pushes you have three ops per push (advance, adjust cfi, offset)

[prime wraith]

yeah i just realized

u dont need to annotate pushes at all, just the final state

i have to do this anyway since ill have a separate function that saves regs

.macro UNWIND_HINT_REG_OFFSET_INC reg
    UNWIND_HINT_REG_OFFSET(\reg, %(\base_offset + offset))
    offset = offset + ULTRA_ARCH_WIDTH
.endm

.macro UNWIND_HINT_GP_REGS base_offset=0
    offset = 0

    UNWIND_HINT_REG_OFFSET_INC r15
    UNWIND_HINT_REG_OFFSET_INC r14
    UNWIND_HINT_REG_OFFSET_INC r13
    UNWIND_HINT_REG_OFFSET_INC r12
    UNWIND_HINT_REG_OFFSET_INC rbp
    UNWIND_HINT_REG_OFFSET_INC rbx
    UNWIND_HINT_REG_OFFSET_INC r11
    UNWIND_HINT_REG_OFFSET_INC r10
    UNWIND_HINT_REG_OFFSET_INC r9
    UNWIND_HINT_REG_OFFSET_INC r8
    UNWIND_HINT_REG_OFFSET_INC rax
    UNWIND_HINT_REG_OFFSET_INC rcx
    UNWIND_HINT_REG_OFFSET_INC rdx
    UNWIND_HINT_REG_OFFSET_INC rsi
    UNWIND_HINT_REG_OFFSET_INC rdi
.endm

i love abusing gas macros since they're crazy

[sterile kayak]

eh if you actually use push instructions you do need to annotate at least the cfa adjustment for every single one

otherwise you don't have the atomicity

[prime wraith]

yeye true

[ornate jasper]

could do that automatically with zig comptime now i think about it, since zig lets you build the string for inline asm at comptime

[wide ether]

i combine this with the actual offsets from the struct itself

this is the advantage you have with inline asm

[prime wraith]

can u show what your iretq frame annotations end up looking like with objdump -W?

000011b8 000000000000006c 0000001c FDE cie=000011a0 pc=ffffffff80008695..ffffffff800086b7
  DW_CFA_def_cfa: r7 (rsp) ofs 56
  DW_CFA_offset: r52 (ss) at cfa-8
  DW_CFA_offset: r7 (rsp) at cfa-16
  DW_CFA_offset: r49 (rflags) at cfa-24
  DW_CFA_offset: r51 (cs) at cfa-32
  DW_CFA_offset: r16 (rip) at cfa-40

I suspect this is bullshit

(this is supposed to be with error code)

Call trace (most recent call last):
    #0 in dump_stack+65
    #1 in vpanic+93
    #2 in panic+82
    #3 in handle_double_fault+14
    #4 in unknown/garbage <0xFFFFFFFFFFFFFFFF>

me when bogus CFI annotations

yeah most likely because 0xFF..FF is the synthetic error code it pushes in this case

so it's off-by-one ig

ah wait im stupid, i just forgot to adjust CFA after saving GPs

still off-by-one somewhere, oh well, will figure out tomorrow

[prime wraith]

nvm it was just a stupid error in how i handle macros

Call trace (most recent call last):
    #0 in dump_stack+65
    #1 in vpanic+93
    #2 in panic+82
    #3 in handle_page_fault+14
    #4 in handle_page_fault_begin+12
    #5 in entry+315
    #6 in x86_entry+46

yay

transparent unwind via an iret frame works!

as per usual i was able to debug it in my brain while falling asleep

so i had to go fix it

better precision than gdb

[gentle sandal]

Does my english fails me or the recent call is first, not last

Also nice! Looks great

[prime wraith]

Yeah ur right, it should say first, probably a brain fart

[wide ether]

python moment

in python traces are FILO

[prime wraith]

never realized that was the case

[wide ether]

yeah, it's odd

[prime wraith]

wait what are u doing in that video

[wide ether]

😭

[prime wraith]

also TIL MCE is considered an abort

so apparently you're not supposed to be able to recover after it

[red parcel]

Isn't there an explicit difference between recoverable and unrecoverable mce?

[prime wraith]

well its defined as an abort so not at spec level at least

yeah idk actually

there's a "corrected machine check error interrupt"

[prime wraith]

ah yeah

[prime wraith]

IDT init... OK

I got reg push/pop first try somehow

[proper tulip]

that's so cool

[twilit geode]

[0.00000000] [INFO] [IDT] Idt init... OK [OK]

[prime wraith]

lol thanks, it's not a lot but i overengineered it with macros etc so it took a ton of effort tbh

and my idt thunks are all 7 bytes padded to 8, stored in a special section so i dont have to create a separate array of function pointers etc

stuff like that

ffffffff80008ca8 <division_error_thunk>:
ffffffff80008ca8:       6a ff                   push   $0xffffffffffffffff
ffffffff80008caa:       e9 c3 fb ff ff          jmp    ffffffff80008872 <handle_division_error_begin>
ffffffff80008caf:       cc                      int3

[proper tulip]

what's that push -1

[prime wraith]

synthetic error code

[proper tulip]

ah i see

[prime wraith]

to unify stack frame

[proper tulip]

i always used 0 for some reason but -1 actually seems like a nicer value for some reason

i think it's more recognizable than 0

[prime wraith]

yeah it actually helped me debug a problem earlier

because i realized i was looking at the error code instead of rip

with 0 its less obvious

[twilit geode]

yeah true

btw what are the plans for ultra

[proper tulip]

"mini linux"

[prime wraith]

read the title post

[proper tulip]

what's your current focus?

[twilit geode]

well I mean like things like allocator, scheduler algo, etc

[proper tulip]

like what are you looking at implementing next

[prime wraith]

i might do lockless log ring next

i have everything needed for it

[proper tulip]

damn i thought you already had some cracked logging stuff

[prime wraith]

nah my logging atm is garbage

also i need to think of how im gonna do init graphs

im not sure yet

basically still at gdt/idt init ok stage

[proper tulip]

do you want to do a DAG based initialization too?

[prime wraith]

probably, because idk how else to handle this problem

like the way linux handles it: roughly separate stuff into initcall levels and depend on linking order

which is hell

[proper tulip]

some people say "just dump all the _init calls into once place"

but i personally don't like this approach as much as just having one piece of code handle it all

[prime wraith]

im not sure how that could work, like some stuff depends on other stuff

and some stuff u want earlier, like logging

and u dont want to spam init_x() calls

[twilit geode]

yeah this is one of the many things I hate about astral

[proper tulip]

that's why DAGs are nice, because you can just list a bunch of dependencies and have it all figured out pretty easily at runtime

[prime wraith]

yeah

[proper tulip]

it's really lightweight too, i think

[prime wraith]

i've looked at how monkuous does it and it looked pretty simple yeah

[proper tulip]

not sure what monkuous does, i'll take a look

[prime wraith]

btw if u wanted to optimze the kallsyms script it's now merged

[proper tulip]

yoo

i mean last time i took a look it seemed pretty optimized already

not sure what more you could do without compromising readability

[prime wraith]

https://github.com/proxima-os/hydrogen/blob/main/kernel/init/main.c#L145 init target stuff here

[proper tulip]

but maybe i missed something

thank

[prime wraith]

yeah im not sure tbh

[proper tulip]

hmm, interesting

oh this does not look like DAG

it looks like a simple list of callbacks

[prime wraith]

yeah

but with dependencies

[proper tulip]

yeah true

i like "managarm initgraph"/DAG more

[prime wraith]

how is it different?

[proper tulip]

you can also do something similar to "init levels" with it too

well, for one it's a proper graph and not a list of things to do with optional dependencies

actually

idk how exactly that changes things but

i like more defined solutions like that

[prime wraith]

hm

[proper tulip]

with monkuous' way of doing thing i think you have to list dependencies manually

and hope nothing conflicts with anything

idk what my point is there tbh

i had something in mind but i lost track

[prime wraith]

lol yeah idk

[proper tulip]

i think this needs further consideration

[twilit geode]

this talk gave me some motivation to actually change this, so now its time to wait for astral to compile

[prime wraith]

[proper tulip]

i think a DAG is more maintainable than a list of tasks + dependencies

especially as the number of targets/tasks grows

[prime wraith]

but u still have to specify dependencies

[proper tulip]

sort of, yes

[prime wraith]

how does it automatically decide?

[proper tulip]

you can create "milestones"

or "targets"

[prime wraith]

same thing in monkuous example, stuff may have sub dependencies etc, no?

[proper tulip]

yeah i guess, idk

[prime wraith]

but thats the same as a no-op dependency ig?

yeah idk

like a phony target

[proper tulip]

yeah, just a target that depends on other targets that doesn't actually do anything by itself

[prime wraith]

i think that technically works in monkuous framework although they dont explicitly do it

[willow fern]

i have no strong opinion about this (yet)

[proper tulip]

i'm trying to find a good reason for DAG other than "future proof" and "it's cool"

but a list of tasks + dependencies might work just fine

they are similar in complexity though, if that's what you are concerned about

using DAG will also make it possible to easily visualize the init process with tools like graphviz :^)

although you can build a DAG from a list of tasks and their dependencies too

[willow fern]

if there is a strong argument for graph based initialisation, i say it is that it means you just specify the dependencies and an appropriate order naturally falls out, and what's more it then becomes amenable to parallelising

[proper tulip]

i am glad someone can put my thoughts into nice words

that's exactly why i prefer DAG based init, no messing around with order because it just works

i wonder how useful the parallelization would be in case of a kernel

would spinning up some thread + waiting for them make it faster than doing things sequentially?

i guess it depends on how complex the tasks actually are

maybe a thread pool would make it worthwile but what's the complexity vs gains

[ornate torrent]

depends (on SMP, task length in time, blocking etc.)

one problem with this kind of thing is that now the initgraph depends on scheduler/process facilities

[proper tulip]

you can just have a parameter to the init function

whether it can use threads or not

then use it to bring up the necessary things without parallelization

which probably isn't much

and then have it do the rest in parallel using threads

[ornate torrent]

seems needlessly complex

[proper tulip]

yes

but so it "mini linux"

[ornate torrent]

IMO something like the following is better in that case:

void start_kernel()
{
    init_early_mm();
    init_early_fbcon();
    printk("mykernel version 1.23.4\n");
    init_early_acpi();
    init_time();
    init_smp();
    init_sched();
    smp_boot_other_cpus();
    dispatch_initgraph_items();
    enable_preempt();
    sched_idle();
}

i.e. init stuff sequentially ("manually") until the scheduler is running across all CPUs, then do rest using initgraph.

[proper tulip]

some people find that cleaner, some people prefer DAG based initialization

ah

well, you don't really need smp for scheduling

but yeah

[ornate torrent]

well you want that if you want to parallellize initgraph across CPUs.

and you probably need to init non early mm as well, otherwise you cannot allocate task structs and task stacks etc..

an argument against initgraph is that either 1) it has no performance advantage over sequential ("manual") init or 2) you need to do a lot of stuff sequentially anyways.

[red parcel]

At work we have something that at compile time goes over all the compiled modules, and does a topological sort to ensure the init sequence

the nice thing about such a system is that you can choose what to compile and it just sorts everything out

And if you forget to compile something it tells you, or if you have a cyclic dependency

[carmine token]

Managarm's initgraph doesn't require allocations, I assume that the other implementations that were inspired by it also don't

the main advantage imho is not that it's faster or anything but that you can specify deps where they actually occur

this is very handy because different targets build different subsystems etc

[carmine token]

so if you do this ^ in reality you have to add a lot of #ifdefs or similar

[ornate torrent]

really? things like very core mm and stuff that is needed to run the scheduler should be mostly the same across different architectures, shouldn't it? and after that point you rely on initgraph anyway.

[carmine token]

on x86 for example, you need to set ups I/O APICs before entering proper multi tasking mode (unless you run your tasks with IRQs disabled)

that in turn requires you to parse ACPI tables

etc

[ornate torrent]

sure

but that's like... really core stuff which you want to do very early.

want a timestamp in printk? well you need a timer for that. but you need ACPI tables for that (HPET to calibrate TSC). and everything wants printk.

also IOAPIC? really?

I can do full multitasking with SMP etc. with only the local APIC.

[uneven abyss]

I prefer the linear init and then dynamic init of devices depending on what drivers you have

coz for things like PCI devices it really does not matter what order you initialize them

neat stuff tho

the bootstages are used in some places

[wide ether]

but it does matter when you scan the PCI bus

[uneven abyss]

wuhduyu mean? like the operation of enumerating PCI devices themselves?

I do that after all this is done

[wide ether]

yes

i would count that as init still

[uneven abyss]

💥

[wide ether]

in menix, everything that happens before /init being called is considered kernel initialization

[ornate torrent]

init implies that it has an end. you may perform PCI bus enumeration during the kernel initialization stage, but you're sure as hell going to keep the device enumeration subsystem operational for hotplug and module loading purposes. arguably, that makes it non-init.

[wide ether]

wut

[uneven abyss]

i am not doing allat

these are hobby kernels not production grade enterprise systems

[carmine token]

How do you do that?

[wide ether]

why not aim high

[ornate torrent]

see current davix. it doesn't even touch IOAPIC.

[carmine token]

You can't really prevent spurious I/O apic interrupts to trigger an IRQ strom if you don't set up I/O APICs before multi tasking

[ornate torrent]

not production grade enterprise

[wide ether]

[ornate torrent]

also hardware bugs don't happen

[wide ether]

they're not real

[ornate torrent]

(is an assumption that most software can make)

[wide ether]

the humble quirk table:

[ornate torrent]

(because it doesn't make sense to try to deal with them)

[wide ether]

you know what you should try running on an SG2042

that thing has IPC bugs

and the ghostwrite vuln

[ornate torrent]

what is the SG2042?

[wide ether]

risc-v 64 core cpu

[ornate torrent]

fake arch

[wide ether]

ghostwrite is funny asf

[uneven abyss]

hardware bugs are a lie made up by Big Cybersecurity to keep selling Cybersecurity

[ornate torrent]

no?

[wide ether]

tldr: you can use a botched implementation of vector instructions to write to physical memory

[uneven abyss]

have you tried the bug before

[obtuse sparrow]

The only hardware bug my system deals with is the F00F bug

[ornate torrent]

can it even be mitigated without loss of functionality?

[wide ether]

no

you have to disable it

[uneven abyss]

I saw it talked about by the lauriewired person

[wide ether]

yes

it's real

[ornate torrent]

LMAO!!!

[wide ether]

https://ghostwriteattack.com/

it's obviously a mistake by the silicon vendor

[carmine token]

spurious I/O apic irqs happen on quite a lot of hw

[uneven abyss]

write a kernel in the userspace of an OS for this CPU by making it exploit the ghostwrite vulnerability

[carmine token]

well, they are not truly spurious but e.g., uefi forgetting to de init some device on a level triggered IRQ

[ornate torrent]

interesting

[wide ether]

user mode linux (kernel mode)

[obtuse sparrow]

And i've been planning to remove the mitagtion for it (even tough my system can run on i586)

[carmine token]

this was the inigraph 4 years ago

[ornate torrent]

(but did you read the project README? baremetal is currently not officially supported )

[carmine token]

by now we have quite a few extra stages

thor: Registering stage generic.fibers-available
thor: Registering stage general.iochannels-discovered
thor: Registering task generic.init-alloc-trace-sink
thor: Registering stage tasking-available
thor: Registering task generic.init-kmsg
thor: Registering task generic.init-reclaim
thor: Registering stage generic.ostrace-available
thor: Registering task generic.init-ostrace-core
thor: Registering task generic.init-ostrace-sinks
thor: Registering task generic.init-profiling-sinks
thor: Registering stage pci.devices-enumerated
thor: Registering task pci.enumerate-dmalog
thor: Registering stage pci.roots-discovered
thor: Registering task pci.enumerate-buses
thor: Registering task smbios.parse-smbios3
thor: Registering task legacy_pc.init-ata
thor: Registering stage x86.cpu-features-known
thor: Registering task x86.enumerate-cpu-features
thor: Registering stage x86.apic-discovered
thor: Registering stage x86.hpet-initialized
thor: Registering task x86.init-boot-processor
thor: Registering stage acpi.tables-discovered
thor: Registering task x86.init-hpet
thor: Registering task x86.discover-apic
thor: Registering task x86.assess-timers
thor: Registering task x86.setup-legacy-pic
thor: Registering task x86.init-rtc
thor: Registering task x86.discover-intel-iommu
thor: Registering stage acpi.fiber-available
thor: Registering task acpi.init-acpi-fiber
thor: Registering task acpi.initialize
thor: Registering task acpi.discover-ioapics
thor: Registering stage pci.bus0-available
thor: Registering stage acpi.ns-available
thor: Registering task acpi.load-namespace
thor: Registering task acpi.boot-aps
thor: Registering task acpi.init-pm-interface
thor: Registering task acpi.init-batteries
thor: Registering task acpi.init-ps2
thor: Registering task pci.discover-acpi-config-io
thor: Registering task pci.discover-acpi-root-buses

[ornate torrent]

Why do you print it like this? It's the same every boot isn't it? Wouldn't it be more interesting to not print it by default (reducing log clutter) but print graph edges if initgraph=debug is passed on the command line?

[carmine token]

yes, that's true

[latent geode]

@prime wraith would you happen to know what the issue is here? I'm trying to link dummy kallsyms file with my kernel object for step 0

error: ld.lld: error: /tmp/.xmake1000/250725/kallsyms/.tmp_ilobilix0.o:(.rodata+0x304): relocation R_X86_64_32 out of range: 18446744071562072064 is not in [0, 4294967295]; references '_text'
>>> defined in ilobilix/kernel/linker-x86_64.ld:27

I just have _text = .; at the start of .text section

[prime wraith]

Can u show the generated c file?

[prime wraith]

Ah wait, just that?

[latent geode]

I'm using linux's kallsyms so it's an .S file

[prime wraith]

Did u use their script?

[latent geode]

yes

[prime wraith]

Can u show what it produced

[latent geode]

it's currently this until I encounter other reference errors

.text : {
    _text = .;
    _stext = .;
    *(.text .text.*)
} :text
_etext = .;

[ornate jasper]

oh fuck i know that error, albeit from completely unrelated places

[latent geode]

yeah one sec

[prime wraith]

yeah that seems right, althought im not sure why u put _etext after the section

[latent geode]

idk :P

linux did it

[ornate jasper]

@latent geode do you discard debug/note/eh_frame/etc in the linker script and if not do you put them somewhere in the kernel area

[prime wraith]

--orphan-handling=error is the best thing

[ornate jasper]

when ive seen that error the most its because of debug/gnu note symbols that get placed in the lower half by default

[ornate jasper]

and/or trying to use symbolic references in my ap trampoline

[latent geode]

I do discard them

[prime wraith]

any chance your BITS_PER_LONG is wrong?

[latent geode]

I just define it to 64

in bitsperlong.h

[prime wraith]

hm

[ornate jasper]

and rodata etc is correct i assume

[latent geode]

I believe so

[prime wraith]

because that looks like its trying to put a u64 into a u32

and the only place u have that is the symbol offset base in kallsyms

[ornate jasper]

.quad is quadword which is 8 bytes, no?

[prime wraith]

thats why i asked this

[ornate jasper]

iirc ive had that error using symbolic references in asm files to stuff in the other half, where the relative offset has to fit in 32 even if its a 64 bit ptr for some reason

[prime wraith]

use the C mode of my script to be more based, maybe that will fix it for u

[latent geode]

I just link with arguments that I would normally use to link with

ld.lld -o <out> ilobilix.o .tmp_ilobilix0.kallsyms.o -nostdlib -static -znoexecstack -zmax-page-size=0x1000 some_libs_here

[ornate jasper]

(in my case it was the AP trampoline trying to symbolically refer to stuff that got messed up by the ap trampoline asm having to be based on 0x8000 but being placed in the binary later)

[prime wraith]

yeah im not sure, it looks correct for the most part

u can try dumping the object file

see what size of a relocation it expects there

[ornate jasper]

the number its saying is out of range is FFFFFFFF80001000 btw

[latent geode]

indeed

[ornate jasper]

which looks a lot like an abs address

[prime wraith]

yeah looks exactly like its trying to store _text in that thing

[ornate jasper]

(really wish that error printed it in hex, would make things easier)

[prime wraith]

and its a u32

[ornate jasper]

yeah, because that sort of thing is meant to be relative but its putting the absolute in there instead

[prime wraith]

this one is supposed to be absolute

kallsyms just adds the lowest symbol address to it to produce the base offset

[latent geode]

I compile it with -mcmodel=kernel
could that have something to do with it? because mcmodel mess ups do throw errors like this

[prime wraith]

i do as well and it works fine

[ornate jasper]

went digging back in message history and my solution dealing with that for my ap trampoline was to add a .org to the trampoline which id forgot

[latent geode]

hmm

[prime wraith]

my C version looks like this

[ornate jasper]

ive had no end of issues in the past with asm trying to use rip-relative for things that make zero sense to use rip-relative for

idk how to control it

so im going to second the suggestion of just try the C output lol

[prime wraith]

https://github.com/UltraOS/Ultra/blob/master/kernel/include/private/symbols.h I even have a header with comments for it

[latent geode]

I hate relocation errors

[prime wraith]

did u dump the object

[latent geode]

what am I looking for in it?

[prime wraith]

send it

[latent geode]

RELOCATION RECORDS FOR [.rodata]:
OFFSET           TYPE                     VALUE
0000000000000304 R_X86_64_32              _text

RELOCATION RECORDS FOR [.debug_line]:
OFFSET           TYPE                     VALUE
0000000000000022 R_X86_64_32              .debug_line_str
000000000000002c R_X86_64_32              .debug_line_str+0x2a

[prime wraith]

R_X86_64_32

i mean yeah

[latent geode]

yes, why is it that?

nvm

[prime wraith]

who knows

[prime wraith]

did u figure it out

[latent geode]

maybe because offset 0x304 doesn't needs more than 32 bits

idk

do I need to tell it that the kernel will be loaded in hh

[prime wraith]

nono thats the offset within the binary

[prime wraith]

you compile asm with the exact same flags as u compile the kernel lol

u just feed gcc/clang the .S file instead of .c

[latent geode]

I do

I have the same flags that I use to compile other .S files in the kernel

[prime wraith]

maybe gas needs to be told with .extern that it's a thing that exists idk

@little aurora why would references to a symbol produce 32 bit relocations in a .S file

[latent geode]

wait I'm so dumb

#ifndef BITSPERLONG_H
#define BITSPERLONG_H

#define __BITS_PER_LONG 64
#define __BITS_PER_LONG_LONG 64

#endif

facepalm

I just looked at bitperlong.h on my system and copied the macro name

it's supposed to be BITS_PER_LONG

[prime wraith]

yeah... thats why i asked

[latent geode]

Sadge

thanks for help

[prime wraith]

Np

[latent geode]

wait I need to do kallsyms for modules too

[prime wraith]

i think for modules linux just uses strtab

[latent geode]

yeah I was thinking of doing that too

modules aren't as big as the kernel

[latent geode]

yes, linux just calls it kallsyms anyway

[prime wraith]

https://elixir.bootlin.com/linux/v6.15.8/source/include/linux/module.h#L390

looks like its just references to module's symtab etc

i mean u must have symtab anyway might as well use it instead of duplicating everything

and since its a tiny number of symbols no one cares

as in linear search would be fast

[little aurora]

depends? how are you building it?

[wide ether]

no

[little aurora]

in general, you should build it with the cc driver and pass the same CFLAGS

as the rest of the project

[prime wraith]

it was a bug in his code nvm

[little aurora]

that includes -mcmodel=kernel and friends

i see

what was the bug?

[prime wraith]

.

[latent geode]

incorrect macro name

[little aurora]

sure

glad you fixed it

[latent geode]

smh my head

[prime wraith]

u probably want to demangle the input

nm has an option for that iirc?

or just pass it through c++filt

[latent geode]

yea

[dusky kraken]

          [-C|--demangle[=style]]

coincidentally -C also means demangle for addr2line, readelf, objdump :^)

[latent geode]

💀

ok I need to increase the size

who would have thought that mangling shortened the symbol name :P

jk

[prime wraith]

why do u have that stuff in your kernel?

[latent geode]

what stuff?

[prime wraith]

std::unique_ptr some net::arp thing lol

did u write all of that yourself?

[latent geode]

not smart pointers

I yoinked that from libstdc++

[prime wraith]

is @system.net like a module?

[latent geode]

yes

[prime wraith]

ah

if i were u i would probably strip everything but the function name

also those dont look like functions i think?

[latent geode]

they are

construct

[prime wraith]

[latent geode]

destroy

I hate this so much

Symbol decltype(_S_construct(fp, fp0, std::forward<std::nullptr_t>(fp1), std::forward<std::nullptr_t>(fp1), std::forward<std::nullptr_t>(fp1))) std::allocator_traits<std::allocator<std::list<std::pair<nostd::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::shared_ptr<vfs::[email protected]>>, std::allocator<std::pair<nostd::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::shared_ptr<vfs::[email protected]>>>>::node>>::construct<std::list<std::pair<nostd::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::shared_ptr<vfs::[email protected]>>, std::allocator<std::pair<nostd::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::shared_ptr<vfs::[email protected]>>>>::node, std::nullptr_t, std::nullptr_t, std::nullptr_t>(std::allocator<std::list<std::pair<nostd::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::shared_ptr<vfs::[email protected]>>, std::allocator<std::pair<nostd::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::shared_ptr<vfs::[email protected]>>>>::node>&, std::list<std::pair<nostd::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::shared_ptr<vfs::[email protected]>>, std::allocator<std::pair<nostd::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::shared_ptr<vfs::[email protected]>>>>::node*, std::nullptr_t&&, std::nullptr_t&&, std::nullptr_t&&) too long for kallsyms (1429 >= 1024).
Please increase KSYM_NAME_LEN both in kernel and kallsyms.c

this is in debug mode btw

[prime wraith]

yeah thats why i dont c++

[latent geode]

in release everything is <512

unnecessary symbols like these are inlined and removed

[prime wraith]

ah

still sucks

[latent geode]

indeed

kallsyms tried to generate .asciz """

which obviously didn't work

[prime wraith]

why lol

[dusky kraken]

interesting to see how modules get mangled

x@y

or rather demangled ig

[prime wraith]

i wonder why the hell linuxes gdt is per-cpu

[proper tulip]

don't you need a per-cpu gdt to have a per-cpu tss?

so you can have per-cpu ist and kernel stacks

[prime wraith]

nope

[proper tulip]

damn what?

[prime wraith]

it's cached so as long as you serialize the loading it doesnt need to be per-cpu

[proper tulip]

so you put the data in the tss entry then ltr

and do so on each cpu

one after the other

interesting

[prime wraith]

well u can guard with a spinlock etc

yeah

[proper tulip]

well yeah of course, that makes sense

i kinda don't like that there will be a stale address in the tss gdt entry at the end

but i guess as long as you don't reload the tss you're fine so eh whatever

[prime wraith]

yeah

i think i realize now how linux per cpu areas work

ill have to implmenet that relatively soon

looks kinda sorta not hard but still a lot of work

basic idea is all per cpu stuff is stored in a special section, that special section is cloned at some offset N times where N is the maximum populatable number of cpus, then to get the clone version each cpu adds its offset to the variable they wish to get their copy of

this is the work horse of it

and ofc each cpu stores its base offset in the gs base thing

this is the dynamically determined base of where the cloned data lives

and then the delta between the section and this address + offset for this cpu is written into per cpu offsets

[sterile kayak]

there are several features linux has that require changing a segment descriptor on each task switch, thus making a cpu-shared gdt incredibly inefficient:

• 32 bit compat mode (you need to change a segment descriptor on each task switch for TLS)
• modify_ldt(2)
• ioperm(2)

[prime wraith]

ah

but isnt ioperm stored in the TSS?

[sterile kayak]

you want to copy an 8 KiB bitmap on each yield?

[prime wraith]

so they make a per-task tss?

[sterile kayak]

i'd assume so

[prime wraith]

I see

[sterile kayak]

because the ioperm bitmap pointer in the tss is 16-bit

so there's no reasonable way to just change the ioperm pointer

and instead you have to switch the tss pointer

[prime wraith]

wait, it's a pointer?

like an offset or something?

[sterile kayak]

yeah it's an offset from the start of the tss

[prime wraith]

ah

[sterile kayak]

the ioperm is [tss_base+ioperm_offset,tss_base+tss_limit]

[prime wraith]

ah

makes sense

also turns out it actually dumps all of that info even in prod builds so thats nice

apparently i have 63 pages of static per-cpu data in my build

[wide ether]

that seems wrong

oh wait this is linux

nvm

[prime wraith]

did u think it was menix

[wide ether]

no

your kernel

[prime wraith]

lol hell no

[wide ether]

why not

the "my build" confused me

menix has like 2 pages of percpu data

[prime wraith]

i have 999 things to implement before i get there

[proper tulip]

that kinda sucks

what (i think) managarm does is they put the per cpu section at the end of the kernel

and when a new per cpu clone is needed they just virtually extend it

[prime wraith]

Well Linux reserves areas for hot pluggable CPUs

[carmine token]

yeah, per cpu gdt is a good choice

Managarm also has per cpu gdts

[azure lynx]

Maestro also has per cpu gdts

[vestal zealot]

I also do per-CPU GDT.

[sterile kayak]

So do I

[wide ether]

same

[twilit geode]

same

[inland wave]

I... did not know using per-CPU GDTs was a thing, is there any benefit for doing that?

[proper tulip]

.

[inland wave]

ohhh, okay

[uneven abyss]

same chain

[ornate jasper]

yallre making me consider it now

also, because i still dont really understand it, what does the number after the name of a man page mean

[inland wave]

same I kinda wanna do it now XD

[ornate jasper]

on the other hand none of the things they use it for are things im planning on having tbh

i dont actually plan to have io port access outside of drivers and the other stuff is 16/32 bit related mostly

[inland wave]

true

[sterile kayak]

if you plan on ever porting wine you'll need modify_ldt

[inland wave]

uhhh, I'll just pretend wine doesn't exist

[carmine token]

that's the "section" of the man page. For example, 2 is syscalls, 3 is libc etc

[latent geode]

same

[prime wraith]

Iirc thats only for compat mode apps

[latent geode]

doesn't wine have a fully 64 bit mode now?

[sterile kayak]

idk, all I know is that wine uses that syscall

[prime wraith]

last time i checked hydrogen it loaded the same one on all cpus with a spinlock, or did i look at the wrong thing

[sterile kayak]

Huh you're right, could've sworn it had a per cpu one

Might've been thinking of a different rewrite

[prime wraith]

lmao

i know your kernel better than u

[sterile kayak]

The rewrites do tend to blend together in my head

[red parcel]

Afaik it only does it for 16bit

There was some discussion about it on flatpak since they disable modify_ldt by default

Tho I have to say, the CVEs from modify_ldt are very funny

[twilit geode]

Yeah, I didnt need the ldt in astral for wine

Though I want to have 32 bit apps at one point

So Ill implement it one day

[prime wraith]

@sterile kayak why do u LLDT when loading the gdt?

[sterile kayak]

The bootloader protocol guarantees nothing about LDTR

I want to make sure there is no LDT

[prime wraith]

ah

what happens if it isnt cleared?

[ornate torrent]

Most likely LDTR pointing into garbage memory. Maybe an attacker could craft a segment descriptor payload and somehow get it into the LDT, then unprivileged userspace can load a segment that it shouldn't be able to.

That would be a massively cool attack tbh.

conforming code segment call gate that jumps to arbitrary code would be fun ;

[sterile kayak]

It'll keep referencing whatever region it was referencing beforehand

This region would most likely be in low memory (userspace)

So it would be possible for userspace to get its hands on it

At which point userspace will be able to add custom segment descriptors and load them

For example, userspace might add a custom call gate that points to userspace-provided code but with the kernel code segment selector

Aka arbitrary ring 0 code execution

[ornate torrent]

oh yeah that's right, I froggot it doesn't point into physical memory directly.

[prime wraith]

@sterile kayak why does your hardcoded gdt not set the 4k granularity bit and also limit values?

[sterile kayak]

because everything's in 64 bit mode

[prime wraith]

are those ignored?

[sterile kayak]

almost everything in a segment descriptor is ignored in 64 bit mode yeah

the only things that aren't ignored are size (16/32/64), present, and type

[prime wraith]

i see

its not really mentioned in the description in the intel manual

probably somewhere else in there

basically just says this

[sterile kayak]

In 64-bit mode, segmentation is generally (but not completely) disabled, creating a flat 64-bit linear-address space. The processor treats the segment base of CS, DS, ES, SS as zero, creating a linear address that is equal to the effective address. The FS and GS segments are exceptions. These segment registers (which hold the segment base) can be used as additional base registers in linear address calculations. They facilitate addressing local data and certain operating system data structures. Note that the processor does not perform segment limit checks at runtime in 64-bit mode.

from 3.2.4 segmentation in ia-32e mode

[prime wraith]

did u derive from this that those bits are ignored?

[sterile kayak]

base is always treated as 0 and limit is not checked

all bits other than those I listed either specify base/limit or modify limit checking behavior

oh and dpl etc isn't ignored ofc

[prime wraith]

alright

thx

[ornate jasper]

note that the base part of the descriptor is still ignored for fs/gs - the fs and gs base in long mode are taken only from the MSRs and the base field in the descriptor is unused

[sterile kayak]

untrue

the MSRs are simply views into the descriptor cache

loading fs/gs manually with mov-to-segreg still writes the base into the descriptor cache from the descriptor

[ornate jasper]

does that zero/sign-extend or just only overwrite the lower bits?

[sterile kayak]

zero extend

[prime wraith]

granularity is still there, and u set it to bytes atm

[sterile kayak]

granularity only modifies the limit and the limit is not checked

therefore it is effectively ignored

[prime wraith]

well ig the manual is wrong there

[sterile kayak]

3.4.4 segment loading instructions in ia-32e mode
Normal segment loads (MOV to Sreg and POP Sreg) into FS and GS load a
standard 32-bit base value in the hidden portion of the segment register. The base address bits above the standard
32 bits are cleared to 0 to allow consistency for implementations that use less than 64 bits.

[ornate jasper]

TIL

[sterile kayak]

it's actually really annoying because you need to save/restore the fs/gs selectors as part of context switches

so if the new thread has a different selector you end up doing swapgs; mov gs, x; swapgs

or something similar

otherwise you fuck up the active (kernels) gs base

[ornate jasper]

well id assume you could use the MSRs instead of the selector, no?

[sterile kayak]

ensuring the base stays correct is separate

the selector can be changed by userspace without kernel intervention and thus it needs to be saved and restored

[ornate jasper]

oh right, dang

[sterile kayak]

otherwise you get uncontrolled ipc and/or context switch detection

[prime wraith]

wait can u elaborate

[sterile kayak]

userspace can load any selector it wants into gs as long as rpl=3 and dpl checks succeed

[prime wraith]

why would u have more than one?

[sterile kayak]

this means that there are at least 2 values in every system that userspace can load into it without issue: null selector with rpl=3 and the selector that sysretq sets ss to

both of these can be loaded into gs by userspace without kernel intervention

[prime wraith]

and so why exactly would this fuck up swapgs?

[sterile kayak]

so if you don't save/restore the segment selectors on context switch userspace has at least one bit of storage per segment register (is it null or the sysretq segment)

[sterile kayak]

if you naively restore gs, i.e. just write the new selector into it on context switch, the cpu will reset the active gs base to whatever the segment descriptor specifies

i.e. userspace can now trigger a kernel null dereference at will

[wide ether]

fun

[wide ether]

i fell for this before

[sterile kayak]

(context switch writes to gs -> gs base gets set to 0 -> next cpu local reference triggers null deref)

[prime wraith]

wouldnt this be a userspace null deref

[wide ether]

no

you are removing gsbase while kernel_gs_base is swapped in

[prime wraith]

hmm

[sterile kayak]

so you have to wrap the write to gs in between two swapgs's or otherwise restore the original gs base

[ornate jasper]

why not restore gs after the swapgs then?

[sterile kayak]

that's also an option but iirc two swapgs's is faster than rdmsr-wrmsr

[wide ether]

that what you're supposed to do

[sterile kayak]

or do you mean the swapgs on kernel exit

that'd work as well I guess

[wide ether]

like, i just read gsbase, update gs, and restore gsbase again

[sterile kayak]

no nevermind it wouldn't because that'd destroy the userspace gs base and so it's still context switch detection

[prime wraith]

https://elixir.bootlin.com/linux/v6.15.8/source/arch/x86/entry/entry_64.S#L965 is this the sort of thing you're talking about?

[sterile kayak]

no that's the handling for interrupts that can be triggered at any time

[wide ether]

just dont allow userspace to set gsbase

[sterile kayak]

including in between the triggering of another interrupt from userspace and the time that interrupt handler does swapgs

i.e. it handles scenarios where interrupted cpl=0 but gsbase belongs to userspace

[prime wraith]

and you're talking about the preemption case?

[ornate jasper]

the problem is that selector setting can bypass that lack of perms

[sterile kayak]

yeah

[prime wraith]

i wonder where that code is

do u have link where u handle that in hydrogen

[wide ether]

wdym

[ornate jasper]

setting the selector overrides the base register and theres always at least two possible options to set it to [as outlined above](#1385970208631427173 message)

assuming im understanding this all right

[sterile kayak]

summary:

• on all sane setups there are at least two values that userspace can write to gs without kernel intervention, so on preemption you have to save/restore the gs selector
• when you write to gs the active gsbase (which during preemption points to the kernel's cpu local area) gets reset to whatever the segment descriptor specifies
• combined: you need to do swapgs-write gs-swapgs or similar to ensure the kernel gsbase stays intact

[sterile kayak]

https://github.com/proxima-os/hydrogen/blob/c18f439c1a8c8f72f14e8eea62ee0d77dc3be7bc/arch/x86_64/kernel/proc/sched.c#L44

also for designs where interrupts are enabled during scheduler operation (e.g. ipl-based systems) you need to disable hardware interrupts while doing this, otherwise a badly timed interrupt might use the wrong gsbase

[prime wraith]

where does this do swapgs u mentioned?

[sterile kayak]

that's the method I used in an old rewrite

the goal is to ensure that at no point are interrupts enabled in kernel or cpu local accesses done while the wrong gsbase is active

one way to do that is to do the gs write in between two swapgs instructions

another is to do what I did here: save the gs base and restore it manually

[prime wraith]

what im confused about is like

why would the kernel gs base get modified, if we're only talking about the user gs base

like if u swapgs, the kernel gs value would get written

and then restored back to null if u swapgs when exiting to user

[sterile kayak]

if you write to gs the currently active gs base gets written to

[wide ether]

• you swap to kernel gs on sc entry
• you update gs for the next user thread
• boom kernel gs is gone

[sterile kayak]

swapgs doesn't swap where it gets the base from, it swaps the base value

[prime wraith]

but u update the user gs?

[wide ether]

only if you don't swap

there is no user gs

[prime wraith]

i mean gs base

[wide ether]

you change gs itself as well

[sterile kayak]

there is no user gs base, only active gs base and a swapped out gs base

if you just write to gs normally the active gs base gets reset

[prime wraith]

but the swapped out remains right

[sterile kayak]

at which point your gs local accesses (aka cpu local accesses) become null dereferences

[prime wraith]

why would i access cpu locals via the user gs base?

[wide ether]

no...

[prime wraith]

i would swapgs after getting into kernel

[sterile kayak]

???

[wide ether]

you swap to kernel gs base when you get into a syscall or interrupt

so you can have cpu local vars

[prime wraith]

i get that yes

why would that value be null

if i just swapped it

[wide ether]

if you THEN update the gs segment when rescheduling, kernel gsbase becomes null

mind you that you reschedule in the kernel usually

[ornate torrent]

• you're running in userspace with user gsbase
• bam! syscall. swapgs happens and you're now in kernelspace with kernel gsbase
• reschedule, needs to write to the gs register (NOT gsbase), zeros out currently active gsbase which is the kernel's
• ya' dead

[prime wraith]

what happens in the third step normally?

[wide ether]

you save current gs base, set the register, and restore the kernel gs base value

[prime wraith]

why doesnt it get zeroed?

[wide ether]

it does get zeroed

[prime wraith]

so whats the diff then

[wide ether]

that's why you create this cool thing called a variable

uint64_t old = rdmsr(GSBASE);
asm("mov whatever, gs");
wrmsr(GSBASE, old);

[sterile kayak]

normal events (gs stays the same):

• swapgs on kernel entry, now active gs is kernel and swapped out is user
• some gs-local (cpu-local) accesses are done
• yield doesn't write gs selector, all is good in the world
• some more gs-local (cpu-local) accesses are done
• swapgs on kernel exit, now active gs is user and swapped out is kernel
  fucked up scenario (context switch needs to change the gs selector):
• swapgs on kernel entry, now active gs is kernel and swapped out is user
• some gs-local (cpu-local) accesses are done
• yield writes to gs selector, active gs base is loaded from GDT (0)
• some more gs-local (cpu-local), uh oh gs base was 0, page fault!
• page fault handler does gs accesses, uh oh gs base was 0, page fault!
• repeat previous step N times
• stack overflow while writing exception frame, double fault!

[prime wraith]

why does yield write to the gs selector in this case?

[ornate jasper]

and if you dont restore the gs register the usermode program can detect that it got context switched?

[sterile kayak]

because the old thread used a different gs selector from the new one

[sterile kayak]

yes

[ornate torrent]

it's a side channel

[prime wraith]

so this assumes u only update gs if needed?

[sterile kayak]

if you always update gs you're always in the second scenario

if you never update gs you have a side channel

so yes

[ornate torrent]

if you don't save and restore gs the register, two processes on the same logical CPU can communicate illegally without the kernel's knowledge or permission.

[prime wraith]

so basically what you were saying is if you update your gs ever during context switching don't forget to update gs base as well?

[sterile kayak]

for reference you need to save/restore every segreg (cs/ss is done for you by the cpu), every segreg you don't do this for is a separate side channel, gs simply has extra considerations because the kernel uses gs relative accesses as well

[vestal zealot]

I just force all seg registers to default settings on ctxswitch because processes shouldn't normally change them anyway

[prime wraith]

sounds like a way to detect preemption also

[sterile kayak]

if you update gs selector during context switch you need to do some extra work to ensure the active gs base will stay the same otherwise your kernel will page fault

[vestal zealot]

True, but is that a problem?

[sterile kayak]

it's not just "oh make sure to also save/restore the user's gs base separately" (though that is necessary as well)

[prime wraith]

right, but in this case it doesnt matter at all whether userspace modified it?

[ornate torrent]

this might still give you some timing-based side channels.

[wide ether]

uname -v

[sterile kayak]

if by it you mean the selector: it matters only because most kernels would omit the restore when unnecessary (because of the extra work)

if you blindly restore the selector it doesn't matter yeah

[prime wraith]

wait so which selector do u use for GS base usually

does it have to be a user data selector?

[ornate torrent]

e.g. if P1 has a primitive to sleep and wakeup in a tight loop causing short times between context switches to and from P2, and P2 has a primitive to detect frequent preemption, P1 might be able to communicate with P2.

[vestal zealot]

I use kernel data in kernel, user data in user.

[vestal zealot]

what

Also couldn't you do this with high-res timers anyway?

[ornate torrent]

it would be cool to play bad apple over context switch timings

[ornate torrent]

yes absolutely

[ornate torrent]

new side project discovered ( I already have so much going on)

[vestal zealot]

My sched time slices are like 0.1 to 1ms long so def noticeable to processes whether they get preempted often

[sterile kayak]

https://elixir.bootlin.com/linux/v6.16/source/arch/x86/include/asm/gsseg.h#L28

call stack:

• native_load_gs_index
• load_gs_index
• loadseg
• x86_fsgsbase_load
• __switch_to
• __switch_to_asm
• switch_to (macro)
• context_switch
• __schedule

[prime wraith]

lol how did u know

[wide ether]

probably looked it up before

[sterile kayak]

nah I just knew it had to be in the call stack of schedule somewhere

so I went down it until I found it

[prime wraith]

is there a technical reason for the userspace to ever update the gs selector at all

[sterile kayak]

nope

[prime wraith]

okay i get it now

[sterile kayak]

not that I know of at least

[prime wraith]

thanks lol

so technically its always 0, even if the user wants thread local storage they dont have to touch it as long as the kernel sets the gs base asked by the user

[sterile kayak]

yeah

[prime wraith]

but u can also if u want for whatever reason do mov rax, ss; mov gs, rax?

[sterile kayak]

indeed

any selector valid for ss is also valid for gs

but null is not valid for ss even though it is valid for gs

[prime wraith]

so strange

[sterile kayak]

therefore there are at least two valid selectors for gs: null, and the one for ss

[prime wraith]

they disallowed non dpl 3 descriptor for ss but for gs u can do 0 as well

like why

cs already specifies the cpl/dpl, why is the data segment needed also

this wouldnt be an issue if they just allowed ss = 0

[ornate torrent]

The usual x86 rationale: because legacy.

[prime wraith]

/* Update the FS and GS selectors if they could have changed. */
if (unlikely(prev->fsindex || next->fsindex))
  loadseg(FS, next->fsindex);
if (unlikely(prev->gsindex || next->gsindex))
  loadseg(GS, next->gsindex);

Also looks like wasted cycles, could've done != instead?

[sterile kayak]

yeah I don't get why they did it like that

like you're already looking at both of them just test for equality

[prime wraith]

yeah lol

maybe something to patch

not that any program touches gs/fs

u have to have two retarded programs running at the same time for this to matter

[ornate torrent]

tasks*

you can have one multithreaded retarded process which does this

[prime wraith]

threads

yeah

[prime wraith]

lol thats a big call stack to walk

i wonder if having that deep of a call stack for such a hot path matters

[sterile kayak]

a lot of it will be inlined

[ornate torrent]

it probably does matter a bit

but yeah, inlining.

[prime wraith]

LTO also ig

[ornate torrent]

Even more stuff could be inlined under LTO, but AFAIK Linux doesn't do that

[prime wraith]

what specifically?

linux does do LTO

[wide ether]

linux also doesn't do -O0

[prime wraith]

or O1 for that matter

[ornate torrent]

oh it does?

[prime wraith]

of course

lol

since 2019 i think

[ornate torrent]

there's CONFIG_LTO or something but it's disabled by default and I don't know if a majority of distributions enable it.

[prime wraith]

it's probably only enabled for when every cycle matters

like cloud setups and stuff

because its undebuggable etc

[sterile kayak]

debugging lto builds really isn't that bad lol

[ornate torrent]

I don't approve of the 'undebuggable' claims because they're really false

[sterile kayak]

gdb works perfectly fine as long as you pass -g

[ornate torrent]

IME it is usually just as easy to debug as -O2

[prime wraith]

well ok with gdb and debug info yeah

[sterile kayak]

it's about as debuggable as regular -O3 imo (maybe -O2 but I never use that so idk)

[prime wraith]

although u will still be seeing a lot of <optimized out>

[ornate torrent]

(and that is without relying on gdb to do any heavylifting for me)

[prime wraith]

u cant really debug a production kernel with gdb, so u will be left with some backtraces that are like 2 functions because everything is inlined

[ornate torrent]

yeah but if you know what <optimized out> means, this is really not that bad.

[ornate torrent]

if everything is inlined, the program counter implies the call stack.

[prime wraith]

iirc optimized out just means the variable is too difficult to fetch and they were too lazy to implement dwarf for it

[ornate torrent]

whenever I see 'optimized out', I am delighted that I wrote code that is obvious enough for the compiler to optimize it

[ornate torrent]

or it has been literally optimized out

[sterile kayak]

no? it means that it has been discarded and the location has been reused for something else

[prime wraith]

depends ig

[prime wraith]

well not always right

[sterile kayak]

almost always

[prime wraith]

anyway id rather see the contents of my variables if im debugging

[sterile kayak]

for that you need O0

[prime wraith]

yeah

[sterile kayak]

like lto will make no difference on how much "optimized out" you see