literally almost impossible
#Zinnia
storm bobcat
near tartan
it's only meant to check that a type exists and isn't forward declared
ASSERT_TYPE(uint64_t) for example
5000iq I'm aware
hybrid island
Wouldn't the compiler simply yell at your for not having the type be complete when you use it
near tartan
yeah ig i just wanted to make archdeps as easy to implement as possible
hybrid island
literally almost impossible
Oh I know how bad C++ parsing gets
near tartan
i want this to run on many isas
hybrid island
You need shittons of semantic information to parse C++
I think if I had to choose a single programming language that should die (at least as it is), it's C++. Despite how much I hate on Java, C++ is clearly way more evil.
hybrid island
C++ is not fine exactly because it is insanely complex
storm bobcat
you dont have to understand it all
near tartan
I'll just say that c++ is poorly designed is all
hybrid island
It also gives us this gem:
hollow hedge
am I the only one who likes this forsaken language 
near tartan
It also gives us this gem:
what
hybrid island
You don't just have rvalues and lvalues like in good old C
No, there's 3 more definitions here because why not
marsh holly
It also gives us this gem:
glUseProgram()
glBindBuffer()
glValue()
glDrawTriangles()
hybrid island
Biggest problem in C++ parsing is probably this here:
identifier identifier ()
This is simultaneously:
- An initializer for a variable
- A function definition
How do you distinguish these whilst parsing? It depends where in the code it appears :)
And what the first identifier is
storm bobcat
I'll just say that c++ is poorly designed is all
its not poorly designed, it's just the result of like 30 years of new features piled onto legacy stuff
well, my point is that it wasnt designed to be this complex
it's just a result of its evolution
near tartan
fair enough
storm bobcat
I do like C++ and i believe if you use it right you can write very good code with it
hybrid island
Yeah but parsing it properly is a complete and utter semantics-dependent nightmare.
storm bobcat
yes
hybrid island
Rust parsing wise seems to be way easier than C++
But harder than C
But for a Rust frontend the parsing isn't the problem, it's the sheer scope of the language
C++ in Lily-CC is impossible on my own because parsing
Whereas Rust in Lily-CC is impossible on my own because borrow checker and trait solver
Who knows, maybe in the distant future there'll be like 10 active devs on it (this will never happen probably) and that would likely be enough to do one or the other of both of these.
near tartan
mrustc it
near tartan
exactly
hybrid island
I'd rather experiment with a custom language frontend
hybrid island
being able to bootstrap rustc is huge still
I'm not interested in Lily-CC existing purely to be a bootstrapping compiler for actual rustc
Maybe I'll go through the effort of implementing just barely enough C++ to compile mlibc successfully
But that would be the extent of it realistically
hybrid island
itd be easier to just write your own libc lol
Yes I am in fact mad that mlibc is written in C++
hybrid island
Writing own libc is not for now
near tartan
yeah
hybrid island
Complete waste of time
We can do it for the coolness bootstrapping points later I guess but you'd still quickly run into a lot of userspace software hard requiring C++
C++ specification as fucking 1800 pages long bruh ts pmo
Band-aid ass "fix" to the ambiguity
storm bobcat
Yes I am in fact mad that mlibc is written in C++
musl then
hybrid island
But mlibc is so easy to port
hollow hedge
Yes I am in fact mad that mlibc is written in C++
oh, huh, I didn't know that [mlibc being written in C++] 
hybrid island
Band-aid ass "fix" to the ambiguity
While this is still bad, this isn't nearly as terrible as I originally thought it would be
near tartan
ig whole zinnia can be made freestanding
hybrid island
With these semantics a naive approach of "try parse declaration but don't consume any tokens if it fails" can work correctly
If followed immediately by "parse expression"
near tartan
the userspace will need c++ regardless for like any other program
even just mesa needs c++
(llvmpipe et al)
hybrid island
ig whole zinnia can be made freestanding
Kernel and servers should be freestanding either way
near tartan
good
hybrid island
They are the host, so they're not hosted.
near tartan
changing that then
I'll move klibc out and repurpose it then
oh right i can actually share it between the kernel and servers
since they'll use the freestanding std*.h files from the kernel as well
hybrid island
Watch my dumbass try making a C++ compiler after all of this shit anyway 
near tartan
hybrid island
Anyway
I just looked at the clock and I made it 4 in the fucking morning again
@near tartan that means it's probably also bedtime for you
I'm going to piss off immediately
storm bobcat
Kernel and servers should be freestanding either way
you can probably figure out a way to use libc in servers tho
near tartan
storm bobcat
I think managarm does it??
hybrid island
I think managarm does it??
no
Managarm has a separate set of libraries for servers
near tartan
yes they do but they dont really use any features of it
only like malloc and shit
posix calls are forwarded to the kernel iirc
which is something i wont be doing
that means servers must be statically linked and have no interpreter
ill write a small set of crts that load the vdso before jumping to main() today
storm bobcat
posix calls are forwarded to the kernel iirc
Yeah I don't like that
near tartan
servers need to be able to run completely detached from the posix world
hybrid island
@near tartan do you ever plan on supporting an arch that doesn't use radix tree paging?
E.g. PPC?
hybrid island
Big-endian as well
hybrid island
Because UVM has pmap be arch-specific
hybrid island
I was thinking something like renaming the current pagetable thing to pmap and relocating it
near tartan
sure
hybrid island
A lot of it can be shared for radix tree paging archs though so I'll see if I can't make that work
near tartan
the main archs i want to support are x86_64, aarch64, riscv64 and loongarch64
any other isas i don't know well enough to reason about them
hybrid island
loong64 is still radix tree paging though right?
lofty copper
loong64 is still radix tree paging though right?
yeah
it has normal-ish page tables, just switched in two (like in arm, I think)
(you can look at my paging code 
)
near tartan
oh damn yeah even for arm we can't just use generic mapping code
lofty copper
But also you can do paging yourself, it has manual page walking
and has an mmu which caches invalid addresses
near tartan
like a negative tlb?
lofty copper
But also you can do paging yourself, it has manual page walking
it gives their instructions that are opinionated on their format
near tartan
skully
hybrid island
So we can't do fully generic pmaps either way
lofty copper
You can
hybrid island
Because they're all just barely different enough
lofty copper
Also it has 4 rings like x86 iirc
But paging does let you set different protections for them
near tartan
rate my klibc
hybrid island
wtf
Also this will compile to memcpy calling memcpy if the compiler doesn't inline for whatever reason
And you still have implicit memcpy happening because the compiler is allowed to call that period.
So any implicit memcpy is all of a sudden a stack overflow
You are like actually allergic to even just reusing what you had before or smth
near tartan
no but i thought it wouldn't optimize as well as using the compiler builtins with AVX and shit
because the existing thing is for the kernel without FPU
hybrid island
Why the FUCK do you want to use AVX in the kernel
hybrid island
Then it shouldn't be called klibc
near tartan
it's not called that
i just said it for some reason
xd
it's called libserver
it has _start and vdso parsing
hybrid island
Even still, I don't think doing this impl for these functions is exactly a great idea.
near tartan
okay then i'll reuse the kernel impl
hybrid island
Go nab them from mlibc or smth
storm bobcat
Even still, I don't think doing this impl for these functions is exactly a great...
yeah __builtin_memcpy calls memcpy
its just gonna stack overflow
storm bobcat
Go nab them from mlibc or smth
writing your own works fine as well, I think the compiler can actually detect the memcpy pattern and optimize as such
hybrid island
writing your own works fine as well, I think the compiler can actually detect th...
It will in fact optimize many copies into memcpy
storm bobcat
if not then rep movsb
median forge
rate my klibc
the real men just copy them from glibc source
storm bobcat
or use simd intrinsics if you want 
hybrid island
the real men just copy them from glibc source
We use an incompatible license
storm bobcat
or use simd intrinsics if you want <:meme:575445094589661215>
i have a project in which i do that
near tartan
yeah __builtin_memcpy calls memcpy
isn't uacpis impl wrong then?
hybrid island
yeah __builtin_memcpy calls memcpy
Not always but it can
Basically, __builtin_memcpy is a compiler intrinsic
near tartan
i thought it just provides its own impl
hybrid island
Here go steal this for a simple working and efficient enough for now impl: https://github.com/badgeteam/BadgerOS-Kernel/blob/main/kernel/src/freestanding/string.c
near tartan
i think my impl is already quite optimized
but ill test it in compiler explorer at some point
hybrid island
Optionally you could add the likes of rep movsb gated behind large size and x86
Don't think compiler explorer is the right benchmarking tool for that
near tartan
hm
hybrid island
I'm sure there's some benchmarking software for memcpy and friends out there
near tartan
flamegraph will be useful at some point
hybrid island
In lieu of that just try the uACPI benchmark with both and see which is faster
near tartan
tru
lofty copper
no but i thought it wouldn't optimize as well as using the compiler builtins wit...
My dumb C memcpy has gotten optimized to use AVX (or SSE if that's not enabled) by llvm
lofty copper
Though the kernel's impl didn't for some reason
storm bobcat
Optionally you could add the likes of `rep movsb` gated behind large size and x8...
i dont think there's a point in gating it
rep movsb is insanely fast
lofty copper
rep movsb is insanely fast
It's not for small copies
near tartan
memcpy_beeeg
hybrid island
i dont think there's a point in gating it
rep movsb is a bit slower than SIMD for the smallest sizes
But yeah anything bigger than maybe 512 bits of data do rep movsb
near tartan
i dont think i ever copy more than a page at a time
storm bobcat
It's not for small copies
really?
storm bobcat
`rep movsb` is a bit slower than SIMD for the smallest sizes
I think simd overall is faster
lofty copper
I think simd overall is faster
Doesn't it make task switching slower? (Though it's kinda inescapable on x86)
near tartan
but you need to save/restore anyways on task switches
storm bobcat
idk probably but I mean in raw performance in userspace SIMD is the fastest thing
hybrid island
I think simd overall is faster
I recall hearing that rep movsb is actually something that the CPUs optimize for
storm bobcat
i just said that
though I think SIMD is still faster for the case of moving 32 bit integers
idk ive measured memcpy vs simd copy on my own graphics code and it was significantly faster
lofty copper
but you need to save/restore anyways on task switches
My kernel was doing that lazily (idr if I changed that or not), but with drivers and servers where you might want to quickly switch to them and go back maybe it's faster to not use SIMD?
lofty copper
idk probably but I mean in raw performance in userspace SIMD is the fastest thin...
I've read an article that loop unrolling and doing several uint64_t copies is as fast
hybrid island
idk ive measured memcpy vs simd copy on my own graphics code and it was signific...
ofc
that's graphics
rep movsb is faster if the copies get proper big though
vast lotus
It's not for small copies
Yeah but the small copies would get inlined by the built-in right?
Only the ones that are variable size and usually small would benefit from having a no-rep-movsb path
lofty copper
Yeah but the small copies would get inlined by the built-in right?
If they have fixed size
vast lotus
For what it's worth, linux's memcpy unconditionally uses rep movsb (if cpuid reports fast rep movsb)
lofty copper
Yeah but the small copies would get inlined by the built-in right?
Yeah, but what I'm trying to say is that there are other ways to do it, which are still as fast
hybrid island
For what it's worth, linux's memcpy unconditionally uses rep movsb (if cpuid rep...
That's in the kernel though
We're discussing in userspace
vast lotus
Oh I missed that part of the conversation then
median forge
For what it's worth, linux's memcpy unconditionally uses rep movsb (if cpuid rep...
how cpuid can report this ???
vast lotus
the same way it reports any feature, there's a bit for it in one of the leaves
i think it's called ERMS or something like that?
near tartan
i wonder what the most elegant way of preventing endianess issues with mmio is
maybe something like
typedef struct { uint64_t val; } be64_t;
uint64_t be_to_host(be64_t value);
hoary cave
how did you go from hating on non-LE to worrying about endianess in a span of a single day
somber solar
lmao
hybrid island
i wonder what the most elegant way of preventing endianess issues with mmio is
Just use htobe and betoh
With a plain integer operand type
near tartan
how did you go from hating on non-LE to worrying about endianess in a span of a ...
i can hate it and still write reusable code, no?
im not saying i will never ever target BE, i just hate it as a concept
LE is better
grave peak
thats why middle endian must return
so that both big and little endian people are unhappy
near tartan
pnp endian
magic charm
maybe something like
```c
typedef struct { uint64_t val; } be64_t;
uint64_t be_t...
That's just as I do
If C had branded types like Modula 3 or typescript I would use those but a struct like this is second best solution
magic charm
With a plain integer operand type
I dislike the lack of type safety in doing so
The compiler can make sure you do things right with these
near tartan
yeah and iirc the ABI is the same as a plain primitive
at least in the sysv callconv
hybrid island
yeah and iirc the ABI is the same as a plain primitive
ABIs actually go out of their way to make this a thing
near tartan
based
hybrid island
(Which is the cause of a known issue in Lily-CC I can't fix right now)
near tartan
rust taught me to create newtypes when the usage of a primitive isn't obvious
modern hamlet
maybe something like
```c
typedef struct { uint64_t val; } be64_t;
uint64_t be_t...
htonl and ntohl
near tartan
no
near tartan
htolol
brisk totem
maybe something like
```c
typedef struct { uint64_t val; } be64_t;
uint64_t be_t...
yeah i think thats the best way to do it
i wrote code for that too except with more clang overloading jank
vast lotus
yeah and iirc the ABI is the same as a plain primitive
on all sane platforms yes but for example i386 returns struct { int x; } in a different way than int
brisk totem
32 bit x86 is shit anyway 🙃
lofty copper
just use c++
vast lotus
the same thing applies to c++
lofty copper
If C had branded types like Modula 3 or typescript I would use those but a struc...
no, I mean for structs
you can do some shit with templates and casting from/to host endian types
so you just use it as a normal int?
near tartan
oh?
near tartan
@hybrid island i did some changes to how initd and posixd are built
they're now freestanding and use a freestanding lib called libserver that contains vdso wrappers, crts, and a subset of libc
so now you can build all of zinnia without a cross compiler
hybrid island
cool
near tartan
hm
this vdso thing is really annoying when trying to implement syscalls
i basically need to change 6 code locations when i modify a call
- headers
- kernel
- vdso
- libserver
- libc
- where it's used
if i remove the vdso i only need
- headers
- kernel
- where it's used
magic charm
maybe you could autogenerate it
vast lotus
aren't libserver and libc "where it's used"
marble salmon
I still don't understand what the advantage of a vdso is other than moving compatibility issues into userspace. But if you modify calls in breaking ways, you don't get that benefit anyway
Other than that, the last 3 things that you listed seem to be the same thing
near tartan
aren't libserver and libc "where it's used"
yesn't
you have the functions themselves and the wrappers that invoke the vdso function pointer
vast lotus
that seems like a very strange way to do it
just include the vdso in the dynamic linker's repository
then you can compile against it as if it's a normal dylib
storm bobcat
I still don't understand what the advantage of a vdso is other than moving compa...
security!!
near tartan
that seems like a very strange way to do it
it's how vdsos are done in linux
near tartan
yeah but I don't have a real dynamic linker in my case
the servers are freestanding static executables
vast lotus
is this code on github? i'd like to take a look
near tartan
most of it is still local, but i can push it to a wip branch rn
vast lotus
resolving symbols from dylibs is 90% of the dynamic linker, so if you have that already turning the existing code into a proper rtld should be relatively easy
marble salmon
90% of the dynamic linker is dealing with ELF / gnu idiosyncrasies like load scope vs global scope
vast lotus
also i'm looking at mlibc's rtld code rn and it looks like at least on the libc side you don't need the wrappers at all since it's already injected into the repository
hybrid island
@near tartan just don't call vDSO functions directly from the servers but only through libserver? I don't see the problem here.
near tartan
libserver doesn't abstract the vdso calls beyond just wrapping the function pointer
near tartan
also i'm looking at mlibc's rtld code rn and it looks like at least on the libc ...
you need to make syscalls to add the vdso to the repository tho
because that allocates memory
marble salmon
ld.so only needs around 10 sysdeps
so it can't be that much work even if it needs to go through function pointers
and monkuous is right that for libc it shouldn't be required
near tartan
it has to
how do you allocate memory for the vdso without syscalls
@vast lotus https://github.com/marv7000/Zinnia/tree/wip
vast lotus
alr i'll take a look
near tartan
src/lib/server/vdso.c
marble salmon
I don't think it's a good idea to let ld.so depend on the vdso
depend in the sense of: doing direct calls into the vdso
It's fine for libc to do that
But for ld.so it just complicates the initialization code path and makes it more fragile for no good reason
hoary cave
but it's really cool when you do it that way
marble salmon
Saving 5 nanoseconds to avoid an indirect call when calling open() from ld.so is a questionable goal
vast lotus
it's not saving any runtime performance since even "directly" calling an imported function is an indirect call through the PLT
for rtld i agree it's not worth the effort
near tartan
but it's really cool when you do it that way <:meme:575445094589661215>
i never said that it's practical or useful
i could just make the syscalls header only, then i don't need to deal with any of this
marble salmon
just use function pointers in ld.so and direct calls to the vdso in libc
I think this is mostly self inflicted pain on your side if you have 3 abstraction layers for a single syscall
near tartan
yeah
hybrid island
Why do you want to hide all your syscalls behind a vDSO anyway?
near tartan
already nuked it
vast lotus
here's a patch to turn libserver into a (limited) dynamic linker: https://hst.sh/uparodocin.diff
it assumes it was linked into an otherwise-static executable that only depends on the vdso, but still
oh and it also includes an unrelated bugfix: _start messed up the stack alignment
near tartan
it did?
vast lotus
yeah
the sysv abi expects the stack to be 16-byte aligned when a function is called
aka the stack is 8 bytes off from being aligned when the function is entered (since the call insn pushes the return addr)
by using jmp to get to __zinnia_entry this push didn't happen and so the stack was erroneously aligned
@@ -43,7 +43,8 @@ void _start() {
#ifdef __x86_64__
asm volatile(
"mov rdi, rsp\n"
- "jmp %c0" ::"i"(__zinnia_entry)
+ "call %c0\n"
+ "ud2" ::"i"(__zinnia_entry)
);
#else
#error "Unsupported architecture!"``` this is the hunk in questioni thought it would already be aligned on _start
ooh wait the abi expects me to call a function
vast lotus
yea
near tartan
makes sense
insane work
bro casually drops a diff "fixed everything btw"
where does vdso_lib come from? do i put libvdso there?
oh wait fuck
it's not built
because it's part of the kernel
vast lotus
that's referencing this var https://github.com/marv7000/Zinnia/blob/80cada1bbffbb13455c96b41510882e7d55d888e/src/kernel/vdso/meson.build#L6
near tartan
bruh
yeah that's disabled if you don't do -Dbuild_kernel
i need to refactor that
or just always build everything
vast lotus
yeah i don't really understand the separation in the first place
near tartan
managarm cosplay
vast lotus
ah
vast lotus
the vdso should be buildable independently from the kernel, you can always make it so that the vdso is unconditionally built
if you don't want to get rid of build_kernel/build_servers
also, i wanted to ask: how is this linked? https://github.com/marv7000/mlibc/blob/ff542cd72cb576e8a8dc0f6e48f3d23a31966572/sysdeps/zinnia/generic/internal.cpp#L18-L19
the vdso linker script removes all zn_* weak aliases so i assume that would result in a linker error
vast lotus
ah ok
was wondering since i had to edit the vdso version script to get linking to work
near tartan
the mlibc that i have here uses the same logic to parse the vdso before it does __mlibc_entry
marble salmon
because thor takes so long to build
Managarm is not split into separate pkgs because thor takes long to build
brisk totem
the vdso should be buildable independently from the kernel, you can always make ...
should it?
marble salmon
But because different parts are built with different compilation settings
vast lotus
i mean should as in "i don't see anything that prevents it", not "it's a good idea"
brisk totem
i mean yes there are situations where it cant be viable
like if you need it for symbols only then ofc you can build it
but if you need to know the addresses of things in the vdso in the kernel (e.g. for syscall trampoline checking) then obv you cant
brisk totem
i can just check if it's inside the vdso map
yeah i mean you might want to know if its in the actually correct entrypoint func
not just somewhere in the vdso
depending on your design you might also need/want to bake the vdso into the kernel
near tartan
i objcopy the vdso into the kernel
marble salmon
what exactly is libserver?
near tartan
smaller binaries
brisk totem
i mean you can just do -Wl,--gc-sections no?
near tartan
doesn't do anything
marble salmon
Not a very convincing argument lol
brisk totem
and -ffunction-sections -fdata-sections
near tartan
already tried
brisk totem
like you shouldnt have very big binaries
near tartan
it's still ~1.5MB for a hello world
brisk totem
did you strip
near tartan
that gets down to 1MB
marble salmon
why not just use mlibc with modified sysdeps?
Maybe we will even do it in the future to run on top of the TBD userspace server manager
near tartan
running servers with mlibc is fine itself, but i don't really see the need
brisk totem
that gets down to 1MB
thats more than musl
so something is very wrong
why is musl 800k even
near tartan
They *are* the host, so they're not hosted.
.
marble salmon
I disagree with that lol
marble salmon
I disagree with that lol
One of the benefits of a microkernel is that servers are not special and you're making them special
marble salmon
You lose a lot by requiring a special environment for servers
brisk totem
One of the benefits of a microkernel is that servers are not special and you're ...
ehhh they are arguably special anyway
lofty copper
here's a patch to turn libserver into a (limited) dynamic linker: https://hst.sh...
Wait, do you run dyn linker in server that starts processes? (I wasn't paying attention, and need to implement dyn linker eventually as well...)
vast lotus
Wait, do you run dyn linker in server that starts processes? (I wasn't paying at...
this is a library that is statically linked into servers
marble salmon
For example on Managarm you can just write servers in Rust or use the STL
vast lotus
it only links itself (and thus the server it is built into)
brisk totem
even if they are just special because they use the native subsystem and not the posix subsystem
vast lotus
and only with the vdso
near tartan
For example on Managarm you can just write servers in Rust or use the STL
yeah but those could just be no_std
lofty copper
For example on Managarm you can just write servers in Rust or use the STL
In pmOS as well 
lofty copper
tbh nostd rust is better anyway
Rust is annoying in general because of all of the potential hidden panics
marble salmon
yeah but those could just be no_std
Chrome could also "just be nostd", just do direct syscalls
That's a stupid argument
brisk totem
imo treating servers specially ends up being useful anyway
marble salmon
Everything "can" be nostd, that doesn't mean that it's useful
brisk totem
but i would also very much write nostd code on linux for system level stuff to make sure you can handle alloc failures
near tartan
Everything "can" be nostd, that doesn't mean that it's useful
the thing is, idk if i would use anything hosted besides maybe malloc from libc
i can imagine that most of the time i'd just end up using direct syscalls anyways
brisk totem
the thing is, idk if i would use anything hosted besides maybe malloc from libc
yeah i think this is like a big thing
tbh a lot of things in the C++ STL arent things i'd want to use either
and the rust std is nice as an OS abstraction, but if you have only one OS its usefulness is not crazy?
brisk totem
tbh a lot of things in the C++ STL arent things i'd want to use either
(like, non-intrusive rbtrees in 2026...)
marble salmon
The usefulness is that you can re-use existing code without NIHing everything
brisk totem
i mean in rust you can do that with no_std!
a huge amount of rust libraries do work no_std if you have alloc
near tartan
if i can get mlibc to not be huge as fuck, i'd use it
marble salmon
In rust you're arguably in a better spot but still
Everything that needs to sleep uses std
brisk totem
The usefulness is that you can re-use existing code without NIHing everything
at the cost of it being slower
marble salmon
Every EBR implementation uses std
brisk totem
Every EBR implementation uses std
EBR?
brisk totem
Almost every async executor uses std
except the embedded ones
brisk totem
Epoch based reclamation
brisk totem
Almost every async executor uses std
yeah but you need your own anyway in practice
because your os primitives likely dont integrate with tokio
marble salmon
You're arguing with theoretical advantages while ignoring practical benefits
brisk totem
and the only one anyone else integrates with is tokio
brisk totem
because your os primitives likely dont integrate with tokio
since wakers are kinda lame
marble salmon
Show me the OS that has actually NIHs this stuff and that did it well enough such that the "my NIH is faster" actually materializes
Your argument breaks down once you consider that rewriting all of tokio/crossbeam/smol/rayon with the same quality of implementation just won't happen
brisk totem
In rust you're arguably in a better spot but still
and in rust the std data structures are actually quite good
i mean doesnt managarm have its own async executor
also if your server needs rayon type concurrency. idk i feel like there are other problems
marble salmon
i mean doesnt managarm have its own async executor
Yes, but out of necessity and not to LARP :^)
near tartan
lol
brisk totem
Yes, but out of necessity and not to LARP :^)
i mean my point was literally that you cant reuse a lot of the normal stuff anyway 🙃
and the stuff you can reuse is often not terribly well written
brisk totem
makes sense
marble salmon
I also never said: drivers should be posix programs, i said: drivers can use a libc with separate bindings
brisk totem
i mean yeah sure
lofty copper
tbh a lot of things in the C++ STL arent things i'd want to use either
But at least you get to handle allocation errors?
brisk totem
But at least you get to handle allocation errors?
i mean yes but the data structures are kinda sucky
and std::bad_alloc can work in OOM but its awfully hacky
and i dont want implicitly checked allocation failures
vast lotus
ah yes the emergency buffers bs
lofty copper
But it's better that "the memory is infinite" and if it isn't just crash
brisk totem
better for what
grave peak
But it's better that "the memory is infinite" and if it isn't just crash
thats a completely valid approach for userspace
unless ur a db or something
lofty copper
but what about microkernels and drivers
brisk totem
imo its not really okay for drivers
grave peak
for drivers yeah
lofty copper
and servers
I don't want something critical to just die because userspace can just die at will
(like yes, there are issues with stack and cow and stuff, but still)
marble salmon
It depends
lofty copper
(like yes, there are issues with stack and cow and stuff, but still)
in the sense that they have to be solved as well, oom safety doesn't come for free
marble salmon
Treating small allocations as infailable in drivers can be fine
If there are other mechanisms to constrain memory footprint
For example, if the allocation size per concurrent request is bounded, you can limit concurrency instead of limiting memory
lofty copper
But still, even if they fail, you can return an error, instead of just dying completely
brisk totem
For example, if the allocation size per concurrent request is bounded, you can l...
yeah but its a lot harder to form hard assurances about that
near tartan
I'm so torn now 😭
near tartan
ig i can do what korona said and make it so rtld uses the function pointer wrapper abstraction, and mlibc proper uses the vdso like a normal dso
but that's also ugly as fuck
marble salmon
Don't you have to do that for posix userspace anyway?
near tartan
yeah im just talking in general now
if servers and posix userspace uses mlibc
ideally i would like rtld to use the same DSO logic like @vast lotus wrote
vast lotus
the self-relocation thing for libserver?
near tartan
yea
vast lotus
it should work perfectly fine for rtld as well
near tartan
if i use mlibc for servers, there's no point in duplicating the logic in libserver
lofty copper
but what was the original issue with not using mlibc for servers?
near tartan
being 1MB in a static build
hybrid island
what
near tartan
even with stripping and --gc everything
marble salmon
Did you debug why this happens?
magic charm
For example, if the allocation size per concurrent request is bounded, you can l...
quite
even some monolithic kernels rely on having certain kernel workers having VM privilege to allocate when all others can't allocate
near tartan
hmmmmmmmmmmmmm
i can't do -static-pie because of rcrt1.o
i can't do -static because of the vdso code
vast lotus
what's preventing you from making mlibc install rcrt1.o
or changing your gcc configuration to not use it
vast lotus
it's just a crt1 suitable for inclusion in static pies
neon flicker
vast lotus
this is the gcc spec fragment that selects the crt1 to use
if your existing crt1 is simple enough and uses rip-relative addressing, you can just make all the various crt1s symlinks to that one
hybrid island
Why are you still insisting on the vDSO despite saying you don't need it
near tartan
jfc
vast lotus
but in any case -static-pie has the same dylib restriction as -static
near tartan
ok but then all of this is kinda pointless
near tartan
Why are you still insisting on the vDSO despite saying you don't need it
because there's 10 different people here telling me 10 different things
and monkuous went through the trouble of writing the patch so i want to at least test it
vast lotus
this is the way i got static/static-pie linking to work in proxima while still dynamically linking the vdso
hybrid island
It's probably best to make a Zinnia servers GCC anyway
vast lotus
basically you have to ensure that the linker command line has -l<vdso> before -static, which to my knowledge can only be done by editing the gcc spec strings
vast lotus
yep
near tartan
why do i bother with anything 😭
hybrid island
BRUH JUST MAKE IT A ONE HEADER LIBRARY
near tartan
yeah
vast lotus
elf toolchains really aren't designed with this use case in mind
hybrid island
And those are just syscall helpers and that's it
near tartan
i hate having public C headers with nonstandard C, but i guess there's no way around it
i cant make a static library and just depend on it anyways, because then libc.a doesn't contain those symbols
hybrid island
i hate having public C headers with nonstandard C, but i guess there's no way ar...
There isn't. You can write an OS in standard C but nobody does.
hybrid island
I have literally not seen a single project that sticks entirely to standard C + assembly
vast lotus
i mean yeah i tested it
near tartan
I have literally not seen a single project that sticks entirely to standard C + ...
nah i mean just in public headers
the impl can have it
it's not that deep
hybrid island
nah i mean just in public headers
WTF are they compiling Zinnia userspace with? MSVC?
Surely not
Lily-CC, clang and GCC (among others) will definitely support the extensions you need
glibc is FULL of GNU extensions
near tartan
autism brain ig
hybrid island
(on a language level that is, library as well)
near tartan
ill change it to header only
near tartan
okay that's done
but now initd is 1mb again
with stripping it goes down to 984KB
without lto there's basically no difference
also tried
'-fdata-sections',
'-ffunction-sections',
'-Wl,--gc-sections',
that doesn't change anything either
hybrid island
How is that a thing
Something in mlibc must be depending on all the functions unconditionally or smth
hybrid island
libc.a might have to be compiled with that
yeah duh
near tartan
galaxybrain
hybrid island
otherwise it'll just collect everything anyway
near tartan
why is that not default
vast lotus
specifically without -fdata-sections -ffunction-sections it'll include everything in every object file that gets included
vast lotus
why is that not default<:nooo:750466870993223750>
probably backwards compat
hybrid island
fair
vast lotus
or some attempt to keep related globals together for cache locality or something
hybrid island
or some attempt to keep related globals together for cache locality or something
No this is just legacy
near tartan
what exactly does the flag do anyways
vast lotus
puts each function/global in its own section instead of just .text/.rodata/.data/.bss
the linker gc works on the section level so that means each function/global can get gc'd individually
for example with -ffunction-sections int foo() would get put in .text.foo whereas without it it'd get put in .text
near tartan
ah, so that requires the default lds to contain something like .text, .text.* right?
vast lotus
yeah
hybrid island
ah, so that requires the default lds to contain something like .text, .text.* ri...
Yeah if you have some symbol you get it in the section .text.main for example
That is what the .* matches in the linker scripts
hybrid island
Well you have e.g.
.text {
*(.text .text.*)
}
That will combine all of them after GC'ing them
near tartan
yeah but not in libc.a
vast lotus
do you not have libstdc++?
near tartan
i do
vast lotus
it should provide cassert
near tartan
oops
copy paste mistake
okay nice
that did the trick
still kinda big tbh
228kb
but this is already much better
nice
marble salmon
what does nm --print-size --size-sort <your binary> | tail print?
near tartan
marvin@aquaduct ~/r/z/build-x86_64 (main)> nm --print-size --size-sort system-root/boot/initd | tail
nm: system-root/boot/initd: no symbols
marvin@aquaduct ~/r/z/build-x86_64 (main)>
before stripping
near tartan
marvin@aquaduct ~/r/z/build-x86_64 (main)> nm --print-size --size-sort pkg-builds/zinnia-servers/servers/initd/initd | tail
0000000000421c80 000000000000135b T _ZN5mlibc7nl_timeC1Ev
0000000000421c80 000000000000135b T _ZN5mlibc7nl_timeC2Ev
0000000000415e80 0000000000001846 T _ZN16ObjectRepository12_parseVerdefEP12SharedObject
000000000041c040 0000000000001c9a t _ZN6Loader23_processLazyRelocationsEP12SharedObject.part.0
000000000041eb00 0000000000002072 T _ZN6Loader13_buildTlsMapsEv
000000000040f4c0 0000000000002275 T interpreterMain
000000000042c348 0000000000002288 r __EH_FRAME_BEGIN__
0000000000432380 00000000000023c0 b _ZN12_GLOBAL__N_1L14startingLocaleE
0000000000434740 00000000000023c0 b _ZN12_GLOBAL__N_1L7cLocaleE
0000000000436b20 0000000000006000 b _ZN12_GLOBAL__N_1L12key_globals_E
marvin@aquaduct ~/r/z/build-x86_64 (main)>
marble salmon
key_globals could potentially use dynamic allocation I guess (?)
not sure why we need both cLocale and startingLocale
near tartan
@fair lintel maybe?
near tartan
I wanted to program for this today <:FeelsBadMan:575445132535398422>
anyways
it cant find _start anymore 🔥
hybrid island
it cant find _start anymore 🔥
lmao
near tartan
🔥
hybrid island
Is this kernel?
near tartan
ye
hybrid island
I'm assuming this is because you're now doing -ffunction-sections and -Wl,--gc-sections in the kernel?
near tartan
i don't use the former in the kernel yet
near tartan
it's plain C 
hybrid island
well anyway my brain is mush gl debugging
hybrid island
inb4 accidentally removed the srcfile
near tartan
💀 wtf
vast lotus
is that the final executable or an object file
near tartan
final executable
vast lotus
i'm guessing that, because it couldn't find the entry point, nothing is reachable from any gc root
near tartan
i was rearranging meson builds and didnt adjust their usages
this used to be klibc_sources =
🗣️
near tartan
servers must not make any posix calls
hybrid island
servers must not make any posix calls
Yeah because that'd obviously explode very quickly
near tartan
ye
but you can still use a lot of functions tbh
let me try malloc
also the entire printf suite
near tartan
How would printf even work
vsnprintf using zn_log
hybrid island
right
For some reason this makes me think of that one guy who made a game that runs entirely using a single printf
near tartan
For some reason this makes me think of that one guy who made a game that runs en...
lo
marble salmon
servers must not make any posix calls
you can just make all posix sysdeps return ENOSYS / not implement them
near tartan
yea that's what i do
hybrid island
That's probably for the best
hybrid island
Make some mlibc compile flag ig
Like it doesn't really make sense using the same exact mlibc for both
vast lotus
yeah you'd probably need a separate mlibc build for that
hybrid island
I wonder how Managarm itself does this
near tartan
mini posix in the kernel
hybrid island
That's... Interesting.
marble salmon
It doesn't really implement posix in the mini posix
It allows a program to open its DSOs and to allocate
We'll probably move this into userspace at some point
hybrid island
It allows a program to open its DSOs and to allocate
That doesn't sound like something that needs special syscalls
marble salmon
wdym by "special syscalls"
hybrid island
Dedicated syscalls just for the mini posix
marble salmon
the mini posix just implements a part of the posix protocol (the part corresponding to what i wrote above) and it fails every other request
near tartan
does mlibc not work with lto?
vast lotus
iirc gcc's lto is somewhat buggy when builtin functions (like memset) are defined in one of the lto'd translation units
near tartan
phenomenal
marsh holly
iirc gcc's lto is somewhat buggy when builtin functions (like memset) are define...
wait so you can't define your own memset and have it be lto? 💀
hybrid island
LTO memset is illegal apparently
vast lotus
idk i never managed to reduce it to an actual minimum reproducible example so i can't be sure that's actually the issue
hybrid island
You could just disable LTO for just the impl of string.h ig
vast lotus
but it's certainly the only type of project i've encountered those types of lto bugs on
marsh holly
damn
hybrid island
Cool I shall DOWNLOAD
And see if I can't build it
After several ID10T errors we compiling image shit again
Do I need a new compiler for kernel+servers now?
Other problem is:
../../../src/ports/mlibc/meson.build:385:22: ERROR: Include dir options/bsd/include does not exist.
I already did a complete cleanbuild
near tartan
you need a cross compiler
for the servers
kernel can still use any freestanding compiler
hybrid island
Do you have the cross compiler for me anywhere?
near tartan
wdym
hybrid island
Patches and shit I need to get the cross compiler
Because it's not the same one as userspace, is it?
near tartan
the kernel is freestanding
hybrid island
What about the mlibc missing include though
near tartan
you should be able to just build it with xbstrap
hybrid island
Wait do I not build the kernel and servers standalone anymore now?
hybrid island
Other problem is:
```
../../../src/ports/mlibc/meson.build:385:22: ERROR: Includ...
Sorry I mean this one
This looks like wrong mlibc
Or something
near tartan
try nuking the ports dir (or just the mlibc dir)
near tartan
Wait do I not build the kernel and servers standalone anymore now?
you can't build the servers standalone anymore
i can show you how to set up LSP in vc at some point too
(for mlibc and servers etc)
hybrid island
that would be good
hybrid island
Because LSP setup is easier
ic
Ok it got past the error
Building GCC once more
near tartan
for reference, it's the same setup as with managarm, except the script is in support/lsp-launcher.sh
hybrid island
Well I haven't worked on Managarm before either
near tartan
there's a handbook
hybrid island
Only promised to do FAT filesystem and promptly forgot
Wait they still don't have it
And that means I still make good on that promise
What
Why are we already cross-compiling CMake?
lmao
hybrid island
-- Installing: /var/bootstrap-zinnia/build/tools/host-cmake/share/cmake-4.2/ That's CMake
Not zstd
hybrid island
Oh this is a host package
hybrid island
Oh god we're building LLVM too
marble salmon
why are you not running it through xbstrap emma?
hybrid island
I really am compiling C compilers all day smh
hybrid island
why are you not running it through xbstrap emma?
I am now
marble salmon
ah
hybrid island
I thought that the servers had to be built separately
but no they're in the xbstrap
thingimajig
near tartan
YOOOO I GOT IPC WORKING
hybrid island
Anyway I'll report back in several hours when LLVM is done 💀
hybrid island
YOOOO I GOT IPC WORKING
nice
near tartan
Anyway I'll report back in several hours when LLVM is done 💀
don't forget about the ram usage
hybrid island
I have 24GiB on this machine that should be just enough
near tartan
ah that's fine
hybrid island
Unless they made LLVM even bigger
near tartan
nah we only compile a bit
hybrid island
Last time LLVM is when I tried (and failed) to add a backend to it
marble salmon
how will zinnia ipc work?
near tartan
rn it's just very simple synchronous message passing
hybrid island
Can I add doors later?
near tartan
yes
hybrid island
That sounds very cool
For doors I was thinking of going with the approach of having the receiver create a thread pool on-demand
Maybe with a system call along the lines of zn_door_accept to accept a single door-call using the current thread
near tartan
FTR i've no idea what doors are or what they do besides they're from solaris
hybrid island
FTR i've no idea what doors are or what they do besides they're from solaris
Basically the idea is that the scheduling unit (called a "sled" in solaris) is capable of switching into the context of another process through a syscall
It has a door return stack in the kernel for, well, returning from door calls.
near tartan
i want to do a lot of shared memory ipc too
i think there's a lot of potential to be had
hybrid island
shmem ipc is much more of a userspace thing
near tartan
yeah
hybrid island
all you should need is a handle type for shared memory, which is easily made
near tartan
and saves you a ton of syscalls and context switches if you can sync it properly
marble salmon
rn it's just very simple synchronous message passing
so you wait until both send and recv are posted and then you copy data between address spaces?
hybrid island
Oh BTW because the VMM is going to be UVM-like, I'd like to see if we can do some cool things like page leasing
Where basically the receiver accepts a CoW view of the memory passed to it
I also had a slightly alternative idea for implementing fork(): Don't have the kernel clone the calling thread's registers, let userspace do that. You need to clone a bunch of handles anyway. This way, we can make the syscall fastpath a lot faster because there is no more need for modifying the registers before returning.
near tartan
if it works, sure
somber solar
Oh BTW because the VMM is going to be UVM-like, I'd like to see if we can do som...
is this a joint project?
hybrid island
TL;DR: For how I'd implement this is that you actually only create a CoW clone of your current memory map, then you create a new process and a thread within with the same old stack and registers.
hybrid island
is this a joint project?
ye
somber solar
nice
hybrid island
I finally got sick of BadgerOS and then Marvin happened to start a microkernel project at basically the same time as me
So now we're here
near tartan
yeah
hybrid island
What was going to be that microkernel instead became the template for the Rust Limine bindings
BTW don't you just love seeing a warning like this come out of LLVM codebase?
/var/bootstrap-zinnia/src/ports/llvm/llvm/include/llvm/ADT/APInt.h:192:18: warning: ‘void operator delete [](void*)’ called on a pointer to an unallocated object ‘1’ [-Wfree-nonheap-object]
192 | delete[] U.pVal;
| ^~~~
yeah but it doesn't pop up in llvm
only in gcc
llvm has many of these in their code
hybrid island
I'm now 4150/5267
near tartan
lfg
hybrid island
compiling LLVM
near tartan
i'm working on something akin to managarm's mbus
though i still don't really understand their code
hybrid island
core contributor
hybrid island
fair
near tartan
the mbus code is pretty cryptic and few docs
as i understand it it's like a telephone operator
near tartan
yeah i want good docs
thinking about ipc design...
i want initd to be the main networking hub
hybrid island
wdym networking
near tartan
not internet networking
hybrid island
Sharing handles to channels etc?
near tartan
like ipc routing
yeah, like if you want to connect to another server in the domain
you would then ask initd to create a connection on your behalf
as in, initd gets special treatment in many aspects of the system
hybrid island
Initd gets to do things other processes need authorization for?
near tartan
yes
hybrid island
hmm
Microkernel FUSE right
Is gonna be much easier
Because all TS is userspace anyway
near tartan
Is gonna be much easier
think so
hybrid island
YOOO XBSTRAP BUILD FINISHED
near tartan
baller
near tartan
Why does empty-image need root
a few distros put these in sbin
same applies to parted and mkfs.*
hybrid island
And what? /usr/sbin is not executable by the user?
hybrid island
That is quite possibly the stupidest reason to need sudo I've heard from you
hybrid island
That's dumbassery
near tartan
idk
hybrid island
I'll get rid of these instances of sudo
near tartan
it's the simplest fix
hybrid island
The way I last built my image is way different and doesn't need sudo
near tartan
and like i said, i guess instead you'd add /usr/sbin/ to PATH
hybrid island
adding sbin to path is just not our problem really
hybrid island
well you also need sudo for losetup right
near tartan
no idea
on opensuse i need sbin to be in PATH so it can find the programs lol
and running as sudo has sbin already in PATH
how did you build your image
hybrid island
Yeah sudo for losetup
near tartan
and the rest?
hybrid island
how did you build your image
I made the filesystem blobs separately then dd them in, the entire GPT layout calculated by a bash script that accepts images and partition types to put in
marble salmon
you can just do PATH=$PATH:/sbin losetup
near tartan
that's what i suggested
marble salmon
well yeah
near tartan
ok so the discussion is pointless anyways 😭
if you need sudo for one command it doesn't matter for the other ones, right?
hybrid island
That's the thing: My image creation doesn't need anything privileged
Yours does
And we could easily re-use mine for this to fix this
somber solar
you can create a disk image without sudo
near tartan
but does yours replicate the mount tree
near tartan
no
the mount tree
like partitions that are mounted somewhere in the rootfs
some files are installed to /boot for example
but ofc that's the ESP, and not the rootfs
hybrid island
I symlink those then replace the symlink with an empty dir just before making the image from the sysroot
marble salmon
if you need sudo for one command it doesn't matter for the other ones, right?
well except that if you make a mistake with any other one it can potentially delete your data lol
somber solar
found the script
I was messing with stuff and wrote it a few months ago
hybrid island
shit
somber solar
oof
hybrid island
I thought that would upload as a file
This here is how I combine the created filesystem blobs
Things that turn a root into a filesystem blob
And the Makefile that combines them
(Marvin this is on the Positron repo, but the others can't see that because not public)
near tartan
this seems much more complex for relatively little reward imo
i agree that i should get rid of the sudos
hybrid island
Actually the reason this is more complex is because having it be too simple caused problems in the past
This way lets me very easily add/remove/resize partitions
somber solar
I'll save a link to those messages in my todo list :P
hybrid island
And I tested it too
So another change I propose is that I bring these here in for image creation so we don't need sudo at all.
near tartan
it might be easier to do this in a python script tbh
then you have less stupid integer handling
hybrid island
Not sure you can copy the blobs in while still having the holes of zeroes
hybrid island
Apparently dd conv=sparse should do it but it isn't for some reason
lmao didn't remove old dd without sparse hang on
Yeah
I will submit a PR with an UPGRADED version of this soon ok?
near tartan
ok
do consider putting this in a python script :3
just because i hate bash and its fucking insane syntax
hybrid island
do consider putting this in a python script :3
Only if you contribute to Lily-CC
Yeah I can do that
Do you wanna VC btw?
near tartan
im in managarm rn
hybrid island
mnaganarm
near tartan
mangam
near tartan
hm
encoding multiple non-fixed-size strings is not the most trivial thing tbh
maybe there's something cool i can do with macros
hybrid island
encoding multiple non-fixed-size strings is not the most trivial thing tbh
That should be up to userspace to figure out tbh
hybrid island
Yeah this is gonna need some utilities very soon
near tartan
and i'm wondering how one could start doing that
hybrid island
And I'm making the pythoM scriBt
near tartan
YOOOOOOOOOOOOOOOOOOOOOOOO
i can create new channels and pass handles over to another process
hybrid island
oh god there's fancy colors how much did you overengineer it
hybrid island
Also I have an image creator pythom
hybrid island
(venv) LinuxBot build$ du -h --apparent-size zinnia.img
4.0G zinnia.img
(venv) LinuxBot build$ du -h zinnia.img
20M zinnia.img
(venv) LinuxBot build$
That also works now btw
Commit time
near tartan
you need to remove old tasks when you delete other files