shkwve: an incomplete kernel/maybe bootloader/UEFI thing/maybe something else if i get bored again | Operating System Development | Page 7

untold basin May 5, 2026, 3:59 PM

#

you create them together

#

well

#

you create a port with a queue attached

rose sonnet May 5, 2026, 3:59 PM

#

My port is a queue

tawdry surge May 5, 2026, 3:59 PM

#

So you create the queue first, right?

#

Ports to queues is many to one

rose sonnet May 5, 2026, 3:59 PM

#

Idk, it's funny to me now that we've basically came up with the similar design independently

untold basin May 5, 2026, 3:59 PM

#

tawdry surge Ports to queues is many to one

yeah

tawdry surge May 5, 2026, 3:59 PM

#

I see

rose sonnet May 5, 2026, 3:59 PM

#

I just have one queue though

#

For me, port is a queue

#

Like 1 to 1

untold basin May 5, 2026, 4:00 PM

#

but you have distinct send rights

rose sonnet May 5, 2026, 4:00 PM

#

Yes

untold basin May 5, 2026, 4:00 PM

#

so its the exact same thing

rose sonnet May 5, 2026, 4:00 PM

#

But they go to the same queue

tawdry surge May 5, 2026, 4:00 PM

#

I guess that can work but you probably need to be able to revoke ports if you design it like that

untold basin May 5, 2026, 4:00 PM

#

but with different names

untold basin May 5, 2026, 4:00 PM

#

tawdry surge I guess that can work but you probably need to be able to revoke ports if you de...

why?

rose sonnet May 5, 2026, 4:00 PM

#

tawdry surge I guess that can work but you probably need to be able to revoke ports if you de...

I do have that ability

untold basin May 5, 2026, 4:00 PM

#

i mean first of all you only need to be told when the last user of a port exits

rose sonnet May 5, 2026, 4:01 PM

#

You can either delete the queue all together, or rewoke rights (from both sides) individually

#

Then also both sides get notified of that

untold basin May 5, 2026, 4:01 PM

#

but yeah also making ports revokable isnt like crazy hard

tawdry surge May 5, 2026, 4:01 PM

#

well yeah but you probably want the server to be able to force termination of a connection if a client misbehaves

untold basin May 5, 2026, 4:01 PM

#

tawdry surge well yeah but you probably want the server to be able to force termination of a ...

how would it misbehave

rose sonnet May 5, 2026, 4:01 PM

#

And all rights get deleted when the port gets deleted

untold basin May 5, 2026, 4:01 PM

#

like send garbage?

#

you just say "no" and fail the request

rose sonnet May 5, 2026, 4:01 PM

#

thonk

tawdry surge May 5, 2026, 4:01 PM

#

Try to spam garbage, for example

rose sonnet May 5, 2026, 4:01 PM

#

Yeah

untold basin May 5, 2026, 4:02 PM

#

im planning to have limits on size anyway

rose sonnet May 5, 2026, 4:02 PM

#

I haven't though of that

untold basin May 5, 2026, 4:02 PM

#

(so that you can presize buffers properly)

rose sonnet May 5, 2026, 4:02 PM

#

I mean there is an easy solution to that

untold basin May 5, 2026, 4:02 PM

#

so like the worst you can do by spamming garbage is dos

#

which you can do in so many other ways

#

its a fifo queue too

rose sonnet May 5, 2026, 4:03 PM

#

tawdry surge Try to spam garbage, for example

The issue with my design is that I just clone the rights, so if one client misbehaves, I'd revoke the right for everyone

untold basin May 5, 2026, 4:03 PM

#

yeah i have that kind of thing too

rose sonnet May 5, 2026, 4:03 PM

#

E.g. in pmbus

untold basin May 5, 2026, 4:03 PM

#

the solution is to just not have to ban individual clients

rose sonnet May 5, 2026, 4:03 PM

#

But it can be easily solved by explicitly asking the clients to create new rights for each client

#

Then it can be more granular

untold basin May 5, 2026, 4:03 PM

#

just dont ban meme

rose sonnet May 5, 2026, 4:04 PM

#

untold basin just dont ban <:meme:575445094589661215>

But this is also useful for when either the client or the server crashes

untold basin May 5, 2026, 4:04 PM

#

if the client crashes no

rose sonnet May 5, 2026, 4:04 PM

#

So you get notified in both of the cases, or like don't have to wait

untold basin May 5, 2026, 4:05 PM

#

if the server crashes that just means the server part got abandoned which is different

rose sonnet May 5, 2026, 4:05 PM

#

As a server, you have to know you have no clients

#

So that you can free the internal state and stuff

untold basin May 5, 2026, 4:05 PM

#

untold basin if the server crashes that just means the server part got abandoned which is dif...

i basically do this: ```rs
let mut ifc = ql.inflight_calls.cursor_mut();
ifc.move_next();
while let Some(node) = ifc.remove() {
let op = unsafe { OpRef::from_raw(node) };
let op = unsafe { op.get_unchecked::<CallTreeq>() };

op.complete_err(uapi::Status::Abandoned);

}

trim seal May 5, 2026, 4:06 PM

#

Never looked into static branches

#

What's the idea there?

untold basin May 5, 2026, 4:07 PM

#

you statically patch a nop into a branch if you decide at runtime whatever condition is met

#

if the condition changes very rarely this makes the common case faster

trim seal May 5, 2026, 4:08 PM

#

Why does that need asm goto tho

untold basin May 5, 2026, 4:08 PM

#

how would you do it otherwise

rose sonnet May 5, 2026, 4:09 PM

#

untold basin i basically do this: ```rs let mut ifc = ql.inflight_calls.cursor_mut(); ifc.mov...

I do this https://gitlab.com/mishakov/pmos/-/blob/main/kernel/generic/messaging/rights.cc?ref_type=heads#L116

trim seal May 5, 2026, 4:09 PM

#

No idea I just learned about its existence

untold basin May 5, 2026, 4:10 PM

#

rose sonnet I do this <https://gitlab.com/mishakov/pmos/-/blob/main/kernel/generic/messaging...

yeah makes sense

#

this is the other side tho

trim seal May 5, 2026, 4:10 PM

#

Alternatives work by patching raw assembly

untold basin May 5, 2026, 4:10 PM

#

trim seal Alternatives work by patching raw assembly

well yes thats the same idea

untold basin May 5, 2026, 4:10 PM

#

untold basin this is the other side tho

this is the receive side (queue) dying

#

i dont notify on a port dying but i can fix that probably

trim seal May 5, 2026, 4:10 PM

#

asm goto just means this assembly can jump somewhere else?

untold basin May 5, 2026, 4:10 PM

#

yes

trim seal May 5, 2026, 4:11 PM

#

Ah

untold basin May 5, 2026, 4:11 PM

#

trim seal asm goto just means this assembly can jump somewhere else?

in a static branch case you do something like ```rs
fn static_bool() -> bool {
asm!("/* patch it into a branch to {a} if it should be taken */ nop", a = label { return true });
return false;
}

rose sonnet May 5, 2026, 4:12 PM

#

The sender side is line 672

trim seal May 5, 2026, 4:19 PM

#

untold basin in a static branch case you do something like ```rs fn static_bool() -> bool { ...

Is this better in if statements because the CPU can see that its a literal and so it doesn't need to predict stuff?

untold basin May 5, 2026, 4:20 PM

#

trim seal Is this better in if statements because the CPU can see that its a literal and s...

its better in if statements because compilers arent dumb

#

they can see that if you have code like if if something { true } else { false } { a } else { b } thats the same as if something { a } else { b }

#

and the return thing is enough for gcc/clang to turn it into a direct branch

#

and yes its better because it avoids a conditional branch

#

and its only a nop if the branch is not taken which is almost always way cheaper

#

e.g. on m1, a nop is 1 uop and you can retire 8 every cycle

trim seal May 5, 2026, 4:23 PM

#

And what if the arch jump helpers are out of line

untold basin May 5, 2026, 4:23 PM

#

a cond branch is also one uop, but its between 0.5 and 1 cycle

untold basin May 5, 2026, 4:23 PM

#

trim seal And what if the arch jump helpers are out of line

in reality its a macro

#

its not actually a function

#

and yes making it an out of line function would make it awful and dumb

untold basin May 5, 2026, 4:23 PM

#

untold basin a cond branch is also one uop, but its between 0.5 and 1 cycle

and its on units 1 or 2 and a nop is on any i think (or none)

trim seal May 5, 2026, 4:24 PM

#

untold basin in reality its a macro

On linux it is a function, although they always_inline it

untold basin May 5, 2026, 4:24 PM

#

trim seal On linux it is a function, although they always_inline it

oh wtf

#

but close enough

trim seal May 5, 2026, 4:24 PM

#

Yeah, see korona msg

#

https://elixir.bootlin.com/linux/v6.6/source/arch/x86/include/asm/jump_label.h#L25

untold basin May 5, 2026, 4:25 PM

#

anyway yeah

trim seal May 5, 2026, 4:28 PM

#

Anyway... microkernel right KEKW

untold basin May 5, 2026, 4:34 PM

#

yeah?

#

hmm copying is actually kinda annoying

trim seal May 5, 2026, 4:36 PM

#

btw are you going to use the new acpi crate

untold basin May 5, 2026, 4:36 PM

#

trim seal btw are you going to use the new acpi crate

rust acpi?

#

lol

trim seal May 5, 2026, 4:36 PM

#

yeah

untold basin May 5, 2026, 4:36 PM

#

im either gonna use uacpi or make my own

trim seal May 5, 2026, 4:37 PM

#

own would be interesting to see

untold basin May 5, 2026, 4:37 PM

#

trim seal btw are you going to use the new acpi crate

guess

#

generic spinlocks are uhhh certainly a design choice

trim seal May 5, 2026, 4:37 PM

#

is that bad?

#

lol

untold basin May 5, 2026, 4:38 PM

#

trim seal is that bad?

well you have no idea what the OS is doing

#

anyway they dont handle allocation failures so its as good as useless for me

trim seal May 5, 2026, 4:38 PM

#

they dont??

untold basin May 5, 2026, 4:38 PM

#

uh is uacpi robust to malloc saying "no u"

trim seal May 5, 2026, 4:38 PM

#

ofc

untold basin May 5, 2026, 4:38 PM

#

trim seal they dont??

lol no

#

they just use the rust std containers

trim seal May 5, 2026, 4:38 PM

#

every single allocation failure is handled gracefully in uacpi

trim seal May 5, 2026, 4:39 PM

#

untold basin they just use the rust std containers

damn

#

will have to talk to the author about it

untold basin May 5, 2026, 4:39 PM

#

they are fine if you dont care about alloc failures

#

and there isnt like a great alternative other than "lol write your own collections"

trim seal May 5, 2026, 4:39 PM

#

...which u kinda do for a kernel crate lol

untold basin May 5, 2026, 4:39 PM

#

which understandably people dont want to do

trim seal May 5, 2026, 4:39 PM

#

yeah it requires a phd in language theory

#

if u wanna write a rust vector

untold basin May 5, 2026, 4:39 PM

#

trim seal yeah it requires a phd in language theory

no its not that bad

trim seal May 5, 2026, 4:40 PM

#

what about linked list or intrusive containers

untold basin May 5, 2026, 4:40 PM

#

there is a crate for that

#

which is like fine

#

i use their rbtree

trim seal May 5, 2026, 4:40 PM

#

ah

untold basin May 5, 2026, 4:41 PM

#

because their linked lists arent fine trl

#

well the linked list iirc is fine but the singly linked list can only push at the front not at the back

#

even tho its trivial to make tailqueues work properly

trim seal May 5, 2026, 4:41 PM

#

huh lol

untold basin May 5, 2026, 4:42 PM

#

untold basin even tho its trivial to make tailqueues work properly

they just dont have one so like

#

but anyway usually linked lists arent great

trim seal May 5, 2026, 4:42 PM

#

so if u wanna push something u have to walk the entire list?

untold basin May 5, 2026, 4:43 PM

#

for the singly linked list yeah

#

because they only have a stack

#

and not a tailqueue

open osprey May 5, 2026, 4:44 PM

#

broadly speaking, what is the architecture of shkwve going to look like?

untold basin May 5, 2026, 4:44 PM

#

open osprey broadly speaking, what is the architecture of shkwve going to look like?

idk it keeps changing every 5 seconds lol

open osprey May 5, 2026, 4:45 PM

#

i can see there are some microkernel like primitives

#

and lots of async

untold basin May 5, 2026, 4:45 PM

#

yeah its a microkernel thing

#

unless i get bored of that in which case it wont be a microkernel

#

and yeah i have some ipc rn and thats it

#

the userspace is extremely barebones

trim seal May 5, 2026, 4:45 PM

#

is it going to be more advanced than managarm

untold basin May 5, 2026, 4:46 PM

#

advanced in what way

trim seal May 5, 2026, 4:46 PM

#

microkernel wise

untold basin May 5, 2026, 4:46 PM

#

no im not going to get a bigger port collection

#

and probably also not more perf

trim seal May 5, 2026, 4:46 PM

#

like the primitives etc

untold basin May 5, 2026, 4:46 PM

#

although maybe

trim seal May 5, 2026, 4:46 PM

#

async model

untold basin May 5, 2026, 4:46 PM

#

its a completely different design so its not really comparable

trim seal May 5, 2026, 4:47 PM

#

can u explain the difference in designs?

untold basin May 5, 2026, 4:48 PM

#

i mean not rly? its just all different i think

#

the only similarity is that we both my kernel and managarm has refcounted handles

trim seal May 5, 2026, 4:48 PM

#

lol

untold basin May 5, 2026, 4:49 PM

#

i think

trim seal May 5, 2026, 4:49 PM

#

isnt it all like

#

send data, get reply

untold basin May 5, 2026, 4:49 PM

#

no

trim seal May 5, 2026, 4:49 PM

#

lol

untold basin May 5, 2026, 4:49 PM

#

trim seal send data, get reply

well first of all lets start with the fact that getting the reply could be an intrinsic part of doing ipc or not

open osprey May 5, 2026, 4:49 PM

#

if i did a microkernel i would certainly use these for ipc

untold basin May 5, 2026, 4:50 PM

#

slow

#

also not async

#

also really complicated to implement

open osprey May 5, 2026, 4:50 PM

#

untold basin slow

they're fast

untold basin May 5, 2026, 4:50 PM

#

sync ipc in general isnt that fast afaik

#

or well it has good latency but not throughput

open osprey May 5, 2026, 4:50 PM

#

doors is

untold basin May 5, 2026, 4:51 PM

#

its sync right?

open osprey May 5, 2026, 4:51 PM

#

fully

untold basin May 5, 2026, 4:51 PM

#

yeah

open osprey May 5, 2026, 4:51 PM

#

it's thread-migrating IPC

#

in place of the thread you have a stack of activations in different contexts

untold basin May 5, 2026, 4:51 PM

#

yeah i get that part

#

but yeah afaiu that has worse throughput

open osprey May 5, 2026, 4:52 PM

#

the principal problem, i spoke of it before, is how it scales to complex, mutually-reentrant activation stacks

untold basin May 5, 2026, 4:52 PM

#

like with my design its not conceptually very hard to allow you to queue up a bunch of operations in one syscall

open osprey May 5, 2026, 4:52 PM

#

stack management becomes difficult

untold basin May 5, 2026, 4:52 PM

#

and you can receive a bunch per syscall trivially

#

and with a doors-style system that cant really work

#

you have at least one syscall per ipc

#

and also your throughput is exactly the reciprocal of the latency (i.e. you couldnt possibly make it any worse except with a worse constant overhead meme )

#

anyway back to lock ordering issues

#

i cant copy while holding the queue lock

#

but i also need to hold the queue lock to prevent cancellation

trim seal May 5, 2026, 5:04 PM

#

Do you have your own UEFI boot proto?

untold basin May 5, 2026, 5:06 PM

#

no

rose sonnet May 5, 2026, 5:14 PM

#

untold basin but i also need to hold the queue lock to prevent cancellation

wdym

untold basin May 5, 2026, 5:14 PM

#

rose sonnet wdym

well you can cancel async operations at any(ish) time

rose sonnet May 5, 2026, 5:15 PM

#

just copy, then check if it's canceled..?

untold basin May 5, 2026, 5:15 PM

#

rose sonnet just copy, then check if it's canceled..?

you cant copy after its cancelled

rose sonnet May 5, 2026, 5:15 PM

#

(except the smp problems)

untold basin May 5, 2026, 5:15 PM

#

because then its a uaf

rose sonnet May 5, 2026, 5:16 PM

#

~~just IPI the other CPU and disable syscalls while copying so it can't be canceled~~

untold basin May 5, 2026, 5:17 PM

#

rose sonnet ~~just IPI the other CPU and disable syscalls while copying so it can't be cance...

well no you cant do that

#

the copying on its own requires taking locks and possibly blocking

#

because you can copy from/to a pager

rose sonnet May 5, 2026, 5:17 PM

#

weren't you copying to kernel's stack first?

untold basin May 5, 2026, 5:18 PM

#

i mean i copy through kernel memory sure

#

i have a percpu for it but yeah

#

but the problem is still there

#

i dont want to reacquire the lock every time i do a little bit of a copy lol

rose sonnet May 5, 2026, 5:18 PM

#

I mean I don't have this problem because my messages hold the buffers

#

while they are in the queue

untold basin May 5, 2026, 5:19 PM

#

yeah but i dont want to do that because thats cancer

rose sonnet May 5, 2026, 5:19 PM

#

it is what it is meme

#

just don't send large messages

untold basin May 5, 2026, 5:20 PM

#

lol

#

anyway the solution is to put a second lock inside of the queue

#

which is what allows you to do cancellation on it

rose sonnet May 5, 2026, 5:21 PM

#

yeah

#

don't you have a lock for rights already?

#

like this is the same issue as if the other side also cancels

untold basin May 5, 2026, 5:22 PM

#

rose sonnet don't you have a lock for rights already?

uh that doesnt help

#

the queue has a lock yes

#

but you cant take that lock while doing a usercopy

tawdry surge May 5, 2026, 5:29 PM

#

trim seal is it going to be more advanced than managarm

How do you become "more advanced" than Managarm regarding its microkernel design? KEKW

untold basin May 5, 2026, 5:36 PM

#

tawdry surge How do you become "more advanced" than Managarm regarding its microkernel design...

^^^^

#

how would you make a more advanced monolithic kernel design anyway

untold basin May 5, 2026, 5:36 PM

#

tawdry surge How do you become "more advanced" than Managarm regarding its microkernel design...

by i getting more performance in every possible scenario due to my incredible design which is totally possible and not at all infeasible

#

well i could add an event that you have to wait on before you cancel

#

i think ill do that

#

hmm except that makes cancelling way more complicated lmao

#

actually better idea: i'll make an event you have to wait on before its actually complete

rose sonnet May 5, 2026, 5:45 PM

#

but what if you rugpull and unmap memory, etc.?

untold basin May 5, 2026, 5:46 PM

#

wdym unmap memory

#

then you get a fault

#

either an error code or just a full exception

rose sonnet May 5, 2026, 5:47 PM

#

wait, do you get async pagefaults in the receiver?

untold basin May 5, 2026, 5:47 PM

#

wdym

rose sonnet May 5, 2026, 5:48 PM

#

I'm confused now to how your ipc works

#

Do both sides have to be calling the kernel at the same time for it to go though?

untold basin May 5, 2026, 5:48 PM

#

no

#

you can asynchronously copy memory?

#

like copy memory from a different process

rose sonnet May 5, 2026, 5:48 PM

#

so your recieve is async

untold basin May 5, 2026, 5:49 PM

#

yes

#

my operations are all fully async except for reply-send which is sync

#

but reply-send is guaranteed to complete without waiting on anything except pagers

rose sonnet May 5, 2026, 5:50 PM

#

but then, who gets the pagefault/error when the async recieve can't happen?

untold basin May 5, 2026, 5:50 PM

#

rose sonnet but then, who gets the pagefault/error when the async recieve can't happen?

noone

#

or well probably the caller idk i havent decided yet

#

i could asynchronously deliver an exception

rose sonnet May 5, 2026, 5:51 PM

#

my receive is synchronous...

untold basin May 5, 2026, 5:52 PM

#

its not?

#

you can have many clients connected to a single recv

#

and you could have outgoing requests also use that afaiu

#

which is like the definition of async

rose sonnet May 5, 2026, 6:00 PM

#

Then I'm confused

#

The things get placed to my queue asynchronously

#

But I do pop front synchronously, and it blocks if there are no messages

#

Doesn't it make it synchronous?

untold basin May 5, 2026, 6:03 PM

#

not really?

#

i mean i guess kinda?

#

but like no id call it async io

rose sonnet May 5, 2026, 6:07 PM

#

I mean I've been thinking about it, and I can make it fully async as an optimization, or whatever

#

Without changing the send API

untold basin May 5, 2026, 6:08 PM

#

rose sonnet I mean I've been thinking about it, and I can make it fully async as an optimiza...

and then what would you do with it lol

rose sonnet May 5, 2026, 6:08 PM

#

(I mean I was also planning to make blocking send)

untold basin May 5, 2026, 6:08 PM

#

make it so that you can receive from ports while you receive from ports?

#

lol

rose sonnet May 5, 2026, 6:09 PM

#

untold basin and then what would you do with it lol

You can significally reduce the number of syscalls?

rose sonnet May 5, 2026, 6:09 PM

#

untold basin make it so that you can receive from ports while you receive from ports?

Yeah

untold basin May 5, 2026, 6:09 PM

#

so batching?

untold basin May 5, 2026, 6:09 PM

#

rose sonnet You can significally reduce the number of syscalls?

you can make a multi-recv op just fine

rose sonnet May 5, 2026, 6:09 PM

#

Or like while doing other work

untold basin May 5, 2026, 6:09 PM

#

thats what i have

rose sonnet May 5, 2026, 6:09 PM

#

I mean sure, but do I need it?

untold basin May 5, 2026, 6:09 PM

#

multirecv is most of the gain imo

rose sonnet May 5, 2026, 6:10 PM

#

rose sonnet Or like while doing other work

Like with rust futures for example, I can just make it set a flag in memory on completion, then in busy servers you'll just check the memory first, before calling into the kernel

#

Like full ring buffer kinda thing

rose sonnet May 5, 2026, 6:12 PM

#

untold basin and then what would you do with it lol

This would remove memory allocations

#

Since it would go into the buffer directly

#

Like what you're doing right now

#

Thinking of which, you can also make the cancelation async trl

untold basin May 5, 2026, 6:13 PM

#

rose sonnet Like full ring buffer kinda thing

sure like completion buffers

untold basin May 5, 2026, 6:14 PM

#

rose sonnet Thinking of which, you can also make the cancelation async <:trl:829316641191886...

not if you want to do rust you cant meme

#

okay now my kernel is getting very confused

#

its saying 0 bytes received

#

and it received data properly

#

i think my syscall code is fucked lmao

#

yeah

#

much better

#

https://paste.sr.ht/~pitust/f16367e6f321167e8fec2ea3f1cc1c9e1b10bf8b

#

userspace works!

#

oh oop its printing a random debug print lol

#

and it works with kasan too albeit more slowly

untold basin May 5, 2026, 9:19 PM

#

im gonna try make gdb work better

rose sonnet May 5, 2026, 9:41 PM

#

who needs gdb

untold basin May 5, 2026, 10:20 PM

#

well i want to make gdb work at least kinda okay

#

instead of it working terribly

#

my plan is to write my own gdb client and server so that i can make a shim that sits between gdb and qemu and lies to gdb about my threads etc

rose sonnet May 6, 2026, 4:38 AM

#

Somehow, I think I never really had problems with gdb

untold basin May 6, 2026, 12:43 PM

#

gdb is the most robust program of all time

untold basin May 6, 2026, 9:12 PM

#

"progress"

#

hardcoded that one

#

hmm

#

pwndbg doesnt really know what is going on it gets the perms wrong too

#

i imagine if i updated to latest it would help

#

lol im making so many errors that are like Data format error: invalid hex string: &{%!m(string=encoding/hex: odd length hex string)}].

#

ok i wrote a hack that makes it work properly

#

time to make it work but using the llvm debugserver's documented api, so that i can actually upstream it lmao

#

because i have a feeling that if i write a PR which calls an api thats like "qsk.plzz_give_memory_map_kthx" they wont really accept that

#

pwndbg-lldb is even better

tawdry surge May 6, 2026, 9:59 PM

#

doesn't lldb support the gdb protocol as well?

untold basin May 6, 2026, 10:00 PM

#

they do

#

thats what you get with the gdbserver protocol lmao

untold basin May 6, 2026, 10:00 PM

#

tawdry surge doesn't lldb support the gdb protocol as well?

thats the only one they support

#

except slightly different because of course it is

tawdry surge May 6, 2026, 10:00 PM

#

ah i thought you meant something else due to:

llvm debugserver's documented api

untold basin May 6, 2026, 10:01 PM

#

yeah i meant those magic extensions

#

lol

tawdry surge May 6, 2026, 10:01 PM

#

ah

untold basin May 6, 2026, 10:03 PM

#

also gdb is so absurdly picky when it comes to the xml you send ngl

#

if you have a union, then supply a size and also start/end values for the fields, you get an error that bitfields must have a specified size

#

which is lol

untold basin May 6, 2026, 10:27 PM

#

python
import pwndbg.gdblib.vmmap
def get_known_maps_v2():
    if not pwndbg.aglib.proc.alive():
        return ()
    pages = list(pwndbg.gdblib.vmmap.info_proc_maps())
    pages.sort(key=lambda page: page.start)
    return tuple(pages)
pwndbg.gdblib.vmmap.get_known_maps = get_known_maps_v2
end
``` okay i have this awful hack

#

which makes pwndbg know how to do proc pid always

untold basin May 6, 2026, 11:22 PM

#

i now have this awesome "flags" thing

#

oh yeah and i have a janky layer system

untold basin May 6, 2026, 11:57 PM

#

and now i even made it so that it can pass through singlestepping

untold basin May 7, 2026, 9:54 AM

#

gdb is so efficient

#

thats why every time you singlestep, it sends 120 packets

#

im not sure how much of it is pwndbg but like cmon

#

i might actually write a memory read cache atp

tawdry surge May 7, 2026, 9:56 AM

#

ig they never optimized single step perf

untold basin May 7, 2026, 10:37 AM

#

i imagine its pwndbg not being efficient

#

and gdb not doing caching at all

#

lldb has multimemread which makes it less bad at least

tawdry surge May 7, 2026, 10:57 AM

#

caching would be quite hard as you'd need to be able to understand all possible side effects of an instruction

#

and there could be insane cases

#

such as memory access trapping to a higher EL with visible side effects meme

vapid echo May 7, 2026, 11:03 AM

#

untold basin gdb is so efficient

Instead of GDB, you could support XNU's kernel debugger protocol which lldb supports

#

idr the name

#

ah KDP

untold basin May 7, 2026, 11:05 AM

#

that means i can only use gdb

untold basin May 7, 2026, 11:05 AM

#

tawdry surge caching would be quite hard as you'd need to be able to understand all possible ...

why? i dont mean caching between instructions btw

#

i mean caching within one instruction

#

it might be pwndbg not gdb idk, but gdb should have caching for this imo. they send many overlapping reads, then send more, and they are very short (even though latency is a bigger cost than throughput here most likely)

tawdry surge May 7, 2026, 11:10 AM

#

ah i see

untold basin May 7, 2026, 11:10 AM

#

im gonna try adding some caching to see how much can be gained but i suspect the answer is "a lot"

tawdry surge May 7, 2026, 11:14 AM

#

Why does it matter though

untold basin May 7, 2026, 11:15 AM

#

its noticably slow (idk if thats the cause but like)

#

because each request is an ipc

#

so if every step you send like 120 rpcs, your perf isnt going to be great

tawdry surge May 8, 2026, 10:43 AM

#

libstdc++'s unordered_map manages to actually lose against the extremely low effort frg::hash_map when testing random keys meme

#

BM_HashMap_Find_frg/1024           6361 ns         6360 ns       110181 items_per_second=161.006M/s
BM_HashMap_Find_frg/16384        212194 ns       212178 ns         3303 items_per_second=77.2181M/s
BM_HashMap_Find_frg/262144      6429989 ns      6428793 ns          110 items_per_second=40.7766M/s
BM_HashMap_Find_std/1024           8952 ns         8950 ns        77770 items_per_second=114.41M/s
BM_HashMap_Find_std/16384        271224 ns       271203 ns         2585 items_per_second=60.4122M/s
BM_HashMap_Find_std/262144      9164705 ns      9162696 ns           75 items_per_second=28.6099M/s

trim seal May 8, 2026, 10:48 AM

#

tawdry surge libstdc++'s `unordered_map` manages to actually lose against the extremely low e...

Does it have the same iterator stability properties?

tawdry surge May 8, 2026, 10:48 AM

#

yes

#

but it doesn't have the bucket API

#

i didn't look into what it takes to impl that because it's stupid anyway

#

the open addressing with linear probing and overflow counters is substantially better in this benchmark (2-3x for small table sizes)

trim seal May 8, 2026, 10:50 AM

#

how can u ensure iterator stability with open adressing

tawdry surge May 8, 2026, 10:50 AM

#

BM_HashMap_Find_rcu/1024           2427 ns         2427 ns       284987 items_per_second=421.954M/s
BM_HashMap_Find_rcu/16384        129493 ns       129479 ns         5392 items_per_second=126.538M/s
BM_HashMap_Find_rcu/262144      4943796 ns      4943186 ns          138 items_per_second=53.0314M/s

#

you allocate once per element

tawdry surge May 8, 2026, 10:51 AM

#

tawdry surge ``` BM_HashMap_Find_rcu/1024 2427 ns 2427 ns 284987 item...

this is for a new rcu enabled table that i'm adding which uses the overflow counters

#

inserts are slightly slower (~15%) though because they need to maintain the counters

untold basin May 8, 2026, 10:52 AM

#

trim seal Does it have the same iterator stability properties?

no

#

ah wait yes

#

but it doesnt iterate in the correct order

#

libstdc++ is like awful though

untold basin May 8, 2026, 10:52 AM

#

tawdry surge inserts are slightly slower (~15%) though because they need to maintain the coun...

are you doing vectorized scanning?

tawdry surge May 8, 2026, 10:52 AM

#

no

untold basin May 8, 2026, 10:52 AM

#

at least use the SWAR trick

untold basin May 8, 2026, 10:53 AM

#

tawdry surge inserts are slightly slower (~15%) though because they need to maintain the coun...

15% is a lot more than slightly ngl

untold basin May 8, 2026, 10:54 AM

#

trim seal how can u ensure iterator stability with open adressing

you can use an indirection vector as well

#

which isnt rcu compatible but is a lot faster

vapid echo May 8, 2026, 10:54 AM

#

untold basin are you doing vectorized scanning?

Then you couldn't use it in a kernel

#

Unless you put it behind openmp or something

untold basin May 8, 2026, 10:55 AM

#

vapid echo Then you couldn't use it in a kernel

theres a trick which lets you compare 8 7 bit values to a single 8 bit value in a few instructions without simd instructions

#

and get a bitmask

vapid echo May 8, 2026, 10:55 AM

#

What is it

tawdry surge May 8, 2026, 10:55 AM

#

then you need to restrict your overflow counters to 255 though

#

ig that's fine

untold basin May 8, 2026, 10:55 AM

#

uhh idr youd have to check the code of any good hashtable

untold basin May 8, 2026, 10:55 AM

#

tawdry surge then you need to restrict your overflow counters to 255 though

folly does this

tawdry surge May 8, 2026, 10:55 AM

#

i assume it rehashes when it's exceeded?

untold basin May 8, 2026, 10:56 AM

#

no

#

they make it sticky if it reaches 254

#

and never decrement it

tawdry surge May 8, 2026, 10:56 AM

#

uh

untold basin May 8, 2026, 10:56 AM

#

and you just hope its rare

tawdry surge May 8, 2026, 10:56 AM

#

meme

untold basin May 8, 2026, 10:56 AM

#

and you make sure your hash function is good

#

if it is, you wont reach it

vapid echo May 8, 2026, 10:56 AM

#

Ah that is what SWAR is

#

I just looked it up

#

I think I used that before

tawdry surge May 8, 2026, 10:58 AM

#

untold basin and you make sure your hash function is good

i said this before but i don't think you can assume that

#

or rather, i don't think a good hash function saves you

#

because no matter how good the hash function is, its range may be quite small

#

compared to the number of insertions/deletions

#

for tables that have small sizes but many updates

vapid echo May 8, 2026, 11:16 AM

#

How do you do fine-grained locking in a hashmap?

#

Is it worth it to do something like sharding or should you do lockfree

untold basin May 8, 2026, 11:22 AM

#

tawdry surge because no matter how good the hash function is, its range may be quite small

wdym

#

like it only produces values in [0, 2^32) or something?

#

the definition of a good hash function is that it produces values which are uniformly distributed over the whole interval required meme

tawdry surge May 8, 2026, 11:23 AM

#

no, it only produces values in [0, 512] because the table is <= 512 entries but receives 1M updates per sec

untold basin May 8, 2026, 11:23 AM

#

ah

#

ahhh yeah i get it

#

yeah maybe it could be problematic

untold basin May 8, 2026, 11:24 AM

#

tawdry surge no, it only produces values in [0, 512] because the table is <= 512 entries but ...

this works way better for folly i think because its effectively a 254*14 entry overflow requirement

#

i think max(true value for overflow refc) = max(probe length), right?

tawdry surge May 8, 2026, 11:25 AM

#

yeah

untold basin May 8, 2026, 11:25 AM

#

so youd need to have a probe length of 254

tawdry surge May 8, 2026, 11:26 AM

#

wait no

#

that's not true

untold basin May 8, 2026, 11:26 AM

#

tawdry surge that's not true

when?

#

i think it is

tawdry surge May 8, 2026, 11:26 AM

#

say you insert into slots 0, 1, 2, 3, 4 w/o collisions and then get a collision on insert into slot 0

untold basin May 8, 2026, 11:26 AM

#

oh right

tawdry surge May 8, 2026, 11:26 AM

#

overflow rc = 1 but proble length = 5

untold basin May 8, 2026, 11:26 AM

#

so its ≤ max probe length

#

since you have overflow if an entry before has a continuous probing run from there to here

#

and you need at least {overflow refc} entries to get it that high

#

so if you have a count of 254 you have an entry for which the probe length is ≥ 254

#

which is already a problem

#

and the solution is to rehash at a load factor that prevents this from happening

#

especially for folly when you have 14-way probing this would be an absurdly bad probe length

tawdry surge May 8, 2026, 11:30 AM

#

yeah true

untold basin May 8, 2026, 11:33 AM

#

but yeah maybe you should rehash once this happens too much

#

maybe rehash every max(4*capacity, 16) insertions or something

#

or every capacity^2 insertions or whatever

#

you want to set the rehash interval to be a multiple of the capacity so that it still runs in amortized linear time

vapid echo May 8, 2026, 12:59 PM

#

@untold basin do you know how folly hashtables compare to abseil?

#

I think abseil has the swiss table design which might only be good with SIMD

#

https://jacksonallan.github.io/c_cpp_hash_tables_benchmark/

An Extensive Benchmark of C and C++ Hash Tables

A comparative, extendible benchmarking suite for C and C++ hash-table libraries.

#

this might be relevant but they don't benchmark folly

#

verstable seems to be quite good

untold basin May 8, 2026, 2:20 PM

#

vapid echo https://jacksonallan.github.io/c_cpp_hash_tables_benchmark/

uhhm using a max load factor of 0.875 with std::unordered_map is certainly one of the ideas of all time ngl

#

also those results uhhhh do not look great

#

like you shouldnt be getting that much spikyness

#

and the string keyed benchmarks are entirely useless because the hash function used is fnv (which is really not great; in particular, for short strings, you kill the performance of all of the open addressed simd tables in many cases)

tawdry surge May 8, 2026, 2:39 PM

#

0.875 is not that insane

#

That's the same that i used above for the frigg rcu table

#

(because 7/8 can be computed w/o division)

untold basin May 8, 2026, 2:44 PM

#

tawdry surge (because 7/8 can be computed w/o division)

yeah thats not the issue

#

the issue is changing the load factor from the default for whatever table you are using is uhhh

#

especially for tables which are using chaining like std thats really bad afaik

vapid echo May 8, 2026, 2:53 PM

#

I think I will implement something like verstable

#

Seems pretty good

tawdry surge May 8, 2026, 2:54 PM

#

untold basin especially for tables which are using chaining like std thats really bad afaik

well, at some point it probably becomes quite bad but at 0.875 the average chain length is probably still quite low

untold basin May 8, 2026, 2:55 PM

#

yeah i imagine its probably not like a huge diff

#

the bigger issue is the string hash is really bad + the results do not look good

#

like i know the stl is bad but its not quite that bad in terms of noise

#

unless you did something really dumb

#

e.g. if you only did one run per sample, or if you use the same input so every sample is the same

vapid echo May 8, 2026, 3:02 PM

#

vapid echo I think I will implement something like verstable

What do you think about this

untold basin May 8, 2026, 3:14 PM

#

it probably doesnt matter

#

unless you do something really dumb like one alloc per object

tawdry surge May 8, 2026, 3:20 PM

#

yeah that variance looks sus indeed

untold basin May 9, 2026, 7:47 AM

#

hmm i found out there are tools similar to gnatprove but for rust

#

and now i wonder if unis have a monopoly on formal verification tools

untold basin Jun 22, 2026, 7:57 AM

#

okay so i have an idea

#

what if i remove refcounting entirely

#

because that would kinda solve a lot of the reference cycle issues i think

#

also i could have userspace explicitly allocate objects if i was careful in the api design

tawdry surge Jun 22, 2026, 7:59 AM

#

you can use capability delegation trees as in seL4

untold basin Jun 22, 2026, 7:59 AM

#

which is nice for accounting

tawdry surge Jun 22, 2026, 7:59 AM

#

but the cost of that is extra book keeping

untold basin Jun 22, 2026, 7:59 AM

#

i was thinking of just giving out rights

#

sort of like a mach send right

tawdry surge Jun 22, 2026, 8:02 AM

#

is that just a weak pointer or how would it work?

untold basin Jun 22, 2026, 8:14 AM

#

tawdry surge is that just a weak pointer or how would it work?

something like that yeah

#

except without preserving the presence of the alloc

#

so a handle is the 3tuple {object ptr, object serial, object rights} (in the kernel ofc)

#

you allocate objects in a virtually mapped region

#

and if the page is mapped there it always contains objects

#

or i might use a global handle indirection table idk yet

#

the handle table approach has some advantages in terms of simplicity

#shkwve: an incomplete kernel/maybe bootloader/UEFI thing/maybe something else if i get bored again