Keyronex

[hexed solstice]

i will get around to doing the ps/2 mouse device and see if things improve when that's in

but that will have to wait

a productive evening though, futexes and threads are in

they might even work

[sinful agate]

I remember debugging that bullshit

Took me and korona 2 weeks of digging through epoll loops

[hexed solstice]

you've got to dig deep for gold sometimes

and now look at managarm

[sinful agate]

Turns out I looked wrong at a glibc function, missed one letter

[hexed solstice]

which one was it?

[sinful agate]

https://github.com/managarm/mlibc/commit/1548c5c8a2e1f4b125d66146378ee06f6b8a8f54

That’s the commit

It’s one god damn letter

[hexed solstice]

oh yeah

[sinful agate]

And I didn’t see it (and neither did korona) until we stepped through the code again

[hexed solstice]

i would've gone nuts trying to figure it out

[sinful agate]

So did I

But it’s fixed so problem solved

This bug would’ve fucked you over btw

Xorg needs this or the main loop message passing thing is completely gone to shit

[hexed solstice]

yeah, it would've been a big issue

as i generally assume mlibc is always doing the right thing

this assumption has served me fine so far

[sinful agate]

It usually does now

But I fucked that function up myself lol

[restive quail]

POLSKA GUROM

[hexed solstice]

bollocks

i can't use xev to debug

"<DEBUG>: In function mbtowc, file ../../../mlibc/options/ansi/generic/stdlib-stubs.cpp:400
__ensure(!"decode_wtranscode() errors are not handled") failed"

one hack later:

so certainly it appears to me that communication between X server and clients is working as expected

trying to launch a second X client, nothing happens, no data transmitted on socket - let's see what's afoot

[sinful agate]

I like your style. If it’s somewhat decent, please upstream

[hexed solstice]

it sets errno to EILSEQ and returns -1 which i think is what it should do

once i have chased this X problem down i will review whether it's proper and if so i will certainly upstream

oh and i also want to make backtrace not immediately panic

i'll figure that one out later

for now i just removed the call to it from xorg

it appears backtrace is called once for initialisation - not to actually prnit a stack trace - so panicking is undesirable in that case

so my current inclination is this: i think maybe i miscoded something in sockfs or unix sockets, some logic is common to all sockets (like accept()) and is in sockfs

as that would explain why it is that x server accepts but one connection

[sinful agate]

You patch the meson.build to make the check return false. That’s how we do it

[sinful agate]

laughs in not implemented

[hexed solstice]

so Xorg server appears to not be blocking on an epoll

let's see what it is blocking on

i see

this is by far one of the best features i ever added

shit!

xev displayed something

xclock displayed something!

[hollow hull]

very cool

[hexed solstice]

twm displayed something!

[hexed solstice]

thank you

your recipes made all the difference

[hollow hull]

thanks i guess lol

[hexed solstice]

make no mistake about it

[hollow hull]

a lot of it is inspired on managarm's recipes (so thanks dennis) but with improvements and fixes

[hexed solstice]

i know you don't have the best feelings about lyre's code, whatever you feel about it, it bloody well worked enough to run Xorg and GTK and suchlike and that was a great motivator for me

aero and managarm were also great motivators

[hollow hull]

thank you. it's fine once i have new-lyre up and running i will make it public

[hexed solstice]

i am really very excited to see it

[hollow hull]

i am going really slowly with this one instead of rushing it

the main difference is that i want to make sure every part of it works 100% up to spec

to make something that doesn't crash after running glxgears for 15 seconds

for some undebuggable reason

but yeah this isn't about lyre

it's about keyronex

getting Xorg working

which is pretty cool

[hexed solstice]

i am very glad to have done it, but i really must give credit where it's due

above all to this community

[hollow hull]

well you're also part of this community

so thank yourself in part

[wet apex]

Yo congrats

[hexed solstice]

cheers, i now join an elusive club to which yourself, mint, the managarm crew, and andy python programmer belong

and posnk

[final hatch]

how hard is it to get Xorg running?

[hexed solstice]

you need unix domain sockets, you need an fbdev (subset) implementation (or your own preferred mechanism to give xorg something to draw on), you may not need threads, but i did them anyway to rule out a potential problem

[wet apex]

You dont need threads for it to draw stuff but I wasnt able to make it get input without threads

[final hatch]

like pthreads?

[hexed solstice]

i noticed it wanted to make pthreads initially but there was a variable one could modify that seemed to turn it off

and libx11 could also be told to nto use threads

[final hatch]

I see

[hexed solstice]

mlibc can implement enough of pthreads for this with just a fork-a-new-thread syscall + get thread id + futex wait + futex wake

[final hatch]

what about creating an X display? like the thing that connects to xorg itself to display stuff? what does that involve?

[hexed solstice]

there is the window manager which draws decorations, Twm built and ran fine just with whatever work i did before to get xorg itself going

Xeyes, xev, and xclock also need no more apparently than xorg itself to do their basic work

[final hatch]

yeah the window manager

what does it take to make one that works well enough

[hexed solstice]

xclock might need setitimer for actually drawing the clock, or at the very least it seems it needs for the clock get function to not just return 0, lmao

[final hatch]

do you know?

[hexed solstice]

oh, writing a new WM? i never wrote one but there is a tutorial at https://jichu4n.com/posts/how-x-window-managers-work-and-how-to-write-one-part-i/ which might be good

[final hatch]

I see... how the hell would a GUI program work if there's no WM?

[wet apex]

I was able to fix it being stuck on midnight on startup by patching one of the calculations tho it still didnt move

[hexed solstice]

they are separate concerns in X11

windows still appear without one, they just aren't, well, managed

[final hatch]

so Xorg takes over by itself?

[zenith karma]

no

btw X11 is the protocol, Xorg is the server

Xorg can draw windows on its own

but you need the wm to render the decorations

[sinful agate]

Nice work @hexed solstice

[final hatch]

the WM renders decorations, and that's it?

does it not handle input, send events related to it, etc?

[zenith karma]

i think it can handle some of the inputs as well

but im pretty sure its like any other app

oh what the wm does is binds to the root window's key handler

[final hatch]

i see

[hexed solstice]

cheers, we truly are moving forward

now there are so many things i need to improve on

threads can't exit. sockets can't be closed and don't do any sanity checking. neither of these were necessary yet

implementing virtual console switching would be a worthy thing too

but it's a time for celebration. i said this 3 and a half months ago

[zenith karma]

oh yeah really nice work btw, i wish i was focused enough to actually get a project this far

[sinful agate]

Something managarm could use as well yeah

[hexed solstice]

cheers, it can sometimes seem endless the things you have to do

[final hatch]

i sometimes do small improvements in nanoshell64 and it takes time

mostly because there's so much to learn

[hexed solstice]

spending time working on a solid VMM and the 'microkernel' functionality was one of the best decisions

those have served me very well

[hexed solstice]

so, a provisional set of todos:

• clean up threads
• clean up sockets
• virtual console switching
• alongside this, expose the keyboard driver such that x11 can grab it
• mouse driver
• PTYs
• port an X11 terminal emulator

i need to investigate how the fbdev driver plays with virtual consoles, maybe it'd be better to enable xf86-video-modesetting (or whatever it's called) and implement drm/kms enough for it, that driver is probably more suited and this also lays groundwork for wayland if i ever want that

[zenith karma]

port kitty

[hexed solstice]

is that Goyal's terminal?

[zenith karma]

yes iirc

[hexed solstice]

i like him

calibre is just downright useful

[zenith karma]

wow i didnt even know he was also behind calibre

[hexed solstice]

and i heard kitty dabs all over alacrity

[zenith karma]

ive heard that too

kitty is a pretty nice terminal emulator last time i used it

[tiny gale]

Kitty is amazing

Extremely lightweight and fast

And very customizable

[stone orbit]

wouldn't all you need in theory be pty support, mesa and python ports?

[brazen smelt]

yes, and fontconfig

[hexed solstice]

i have fontconfig and python

ptys are on the way, just like reform in south africa

mesa in principle, i don't imagine it would be intractable

[tiny sapphire]

yo!!!

congrats!

[hexed solstice]

cheers, and many thanks for aero helping light the way

[hexed solstice]

working on mouse input

xorg server seems to poll it during startup but not thereafter

it no longer includes it in its poll set

i suspect an issue in my futexes maybe

let's check if it happens w/ input thread disabled

well, it's certainly a threading issue, now i can do this with threading disabled

more problematically there appears to be some race condition in the slab allocator activated by very extensive TTY activity. i will investigate that later

in general doing much in this environment triggers all sorts of issues

xeyes remains not to work properly

bizarre, i will have to investigate

[vale pelican]

looks funky

[hexed solstice]

yeah, i am not rightly sure as to why it would behave like this

stability is much improved now that one issue is addressed (i prepared a generic ring buffer for use in devices which take the form of "read out some fixed-size entries of a ring buffer" but didn't change sizeof() used in the previous nongeneric byte-only ringbuffer to elementsof(), which led to the PS/2 mouse overwriting its own interrupt object, causing a thankfully predictable but bizarre crash that happened very quickly)

another oddity

when i drag windows they drag only on one axis

it appears if the predominant mouse movement at the onset of dragging is along the X axis it drags along the X, in the opposite case along the Y

[final hatch]

it looks like it's sharing the buffer from Event Tester to xeyes

its really strange

[hexed solstice]

drag anything over it will induce this, e.g.

[final hatch]

so I guess it's a failure to refresh properly?

[hexed solstice]

this is the odd part

[final hatch]

maybe xeyes doesn't handle repaint requests properly

[hexed solstice]

i put code in xeyes to spam my console with information whenever it draws

and it does draw

or at least it claims to

[final hatch]

xorg is probably messed up then

[hexed solstice]

it may be a timer issue

i don't think i return an accurate time atm

and i see some logic in xeyes relating to timeouts

if i comment out #define PRESENT 1 (enabling use of X Present extension) in Xeyes it works

[final hatch]

meanwhile I'm stuck investigating my setjmp and longjmp implementation since I suspect these are the causes of a bug

(which is actually not mine and I stole from somewhere but I thought it worked well right up until it didn't)

[final hatch]

okay well nvm this is a trivial bug, I was not saving the registers properly 🤣

it's as simple as "I don't back up the non scratch registers ebx, esi, edi"

[hexed solstice]

it's always satisfying to solve a simple issue and get a fun result

[hexed solstice]

and now i really mean it when i say: i should take a break from rapidfire unpolished, poorly tested feature additions to support fun ports, and make robust the stuff i already have

[hollow hull]

very nice

[hollow hull]

i have been victim to this

which is why i will take my sweet time with new lyre

to make sure everything i add works to a T

before moving on

[hexed solstice]

yeah, what is noteworthy to me is that i have had virtually no grief from the dispatcher logic (handles kernel threading, synchronisation) nor the VMM because i carefully designed those and implemented them, if not to spec, at least to a fairly comprehensive plan on how they work, where i tried to address all possibilities

but in other areas all sorts of minor bugs pop up

it is a testament to the power of taking one's time that i've had virtually no problems from these components. but from other components, various bugs

and cheers, i am proud to have gotten keyronex to this state, even if i haven't been writing the most excellent code in the rush towards it

i think it says much that the projects of our members so often achieve results like this

[sinful agate]

cries in my workflow

[brazen smelt]

to a T?

wdym?

[hexed solstice]

perfectly well

[brazen smelt]

ohh

[hexed solstice]

i think i am having some kind of futex problem

not immediately apparent what

unfortunately this isn't the time to investigate it

oh i see

i wasn't returning the TID from the clone function

that's better

anyway it's time for sleep

[hexed solstice]

o woe, misery, and terror

i forgot to design epoll for multithreaded use

(at least that's why i think i am having trouble with epoll when xorg with input thread is built)

[hexed solstice]

never mind, i have bigger issues, horrible issue running without accelerated qemu

depending which breakpoints i set different functions fault

ugh

it's completely beyond me what's going on

i need execution replay

oh fuck off

it works fine with execution replay saving

great, i managed to make it crash in the asme way with execution replay recording

now i need to figure out how to reverse step

[hollow hull]

i never managed to make execution replay work

maybe i'm stupid or something lol

i just ended up using bochs for those kinds of heavy debuggings

i plan on avoiding the need altogether in new lyre

[hexed solstice]

https://www.qemu.org/docs/master/system/replay.html#snapshotting-label it's all greek to me

what's all the crap about disk drives

i just want to save execution and step back through it with gdb

[hollow hull]

yeah it's really insistent about the disk drives thing

[hexed solstice]

i thought i'd also avoid the need but suddenly execution is jumping to rip = the weirdest stuff

[hollow hull]

it is frankly annoying when that happens

i can 200% relate to that

i had bugs like that in old lyre that i just gave up on debugging

[hexed solstice]

"qemu-system-x86_64: Device 'rr' is writable but does not support snapshots
qemu-system-x86_64: Could not load snapshot for icount replay"
wtf is this crap

[hollow hull]

because replay doesn't work and getting to the issue with bochs is pretty much impossible

[hexed solstice]

bochs is too special to me

i have no idea how to work it and learning now would be more effort than i care to embark on

[hollow hull]

it's not too complicated

it has helped me in the past with similar jumps to fucktown bugs

[hexed solstice]

it probably is for me, you don't know how dense i am

but honestly it might be necessary to learn as i just can't figure out how the hell to use qemu replay

[hollow hull]

i don't think anyone in this server has successfully managed to use it other than just pointing out its existance

i tried but gave up after a day of trying to figure it out

mainly because i think my conclusion was that it was not something doable

a paper feature so to speak

[hexed solstice]

yes, i think you might be right

i've put in the incantation

qemu launches and stays on "guest has not initialized the display yet" at address 0x7e04

gdb commands like continue, stepi, etc do not work

this is really annoying

they advertise this feature as usable but it's a myth as far as i can tell

what i wonder is whether this netdev crap you have to do for it has anything to do with it

maybe it exports some special protocol on that netdev

i mean, gdb has the commands

maybe i need to re run and generate those crapping snapshots

the documentation is absolute trash anyway

[hollow hull]

it is

if it is anything other than a paper feature then it is documented beyond horribly

[hexed solstice]

whatever it is it's really fucking ugly

it doesn't make the blindest bit of sense that it needs -net and -drive and -device ide-hd,drive=img-blkreplay to do this

this has really wound me up

i've wasted my time on this nonsense and i still have a godawful bug which i simply cannot reproduce because single-stepping seems to inhibit it

[hollow hull]

after dinner i may look into either patching qemu or adding some ugliness that allows you to put in a "bad" IP and it will execute until the "bad" IP is hit and show a trace of the last IPs leading to the bad one

it sounds like a dearly needed feature

[hexed solstice]

that would be exceptionally useful to me

as this problem is completely eluding me

[hexed solstice]

i have no idea what to do or where to goh ere

this bug is beyond me

i really don't know if i'll be able to pin it down without reverse execution. it could be anything

[hexed solstice]

i will give up for the night shortly

this is what i get for not regularly testing without acceleration

[hexed solstice]

giving up for the night

[hexed solstice]

if you do this give me a shout

otherwise i might be forced to do it myself because i am truly lost

or even just a constant stream of IPs

[hollow hull]

im busy playing poker in vc

later i will give it a look and let you know

[gentle hatch]

Git bisect could help if this really is an old bug

[hexed solstice]

it might if i could figure out how to build my old versions

i can barely remember how to build the current version

[gentle hatch]

lol

[hexed solstice]

it's probably what i'm going to have to turn to

since i simply have no clue what's happened

well, the problem isn't present in some older version

it was introduced in the last 18 days then

that's helpful

[brazen smelt]

this sounds like a bug I had once ago

where a parked thread’s context would get changed, then the thread would resume with a garbage context

I have a feeling like you’re encountering a scheduler bug, do you think that could be the culprit?

[hexed solstice]

i inserted a test to determine whether the thread switch ever tried to switch to a thread whose saved IP was absurd and it only triggered once, which was when the thread had already gotten to a funny IP but was preempted before it did any funny business

[hexed solstice]

Noting here to check when I get home whether, when I introduced ASTs, I might have accidentally interfered with the interrupt priority mechanisn

[hexed solstice]

confirmed

i broke the interrupt dispatch function such that it lowered IPL to 0 when an AST came in or when a TLB shootdown was requested

now keyronex boots to bash in unacccelerated qemu

[hexed solstice]

fine. i'm happy with this

the bug i feared would take me forever to find is, touch wood, gone after less than 24 hours

unfortunately it really took the wind out of my sails on what i was previously doing, which was trying to solve various less annoying bugs (there is a null pointer dereference in relation to process groups/sessions from time to time) and significantly improving the quality and completeness of existing code

[hollow hull]

well i guess i don't have to deal with that qemu thing anymore then

[hexed solstice]

unless it's something that really takes your fancy, many thanks for looking into it regardless

i should thank @gentle hatch for prompting me to actually try building an old keyronex

because discovering that older keyronex worked fine, i ended up figuring out the likely cause of the bug simply when i had a casual flick through the commit log since that date on github during break

[hexed solstice]

pgroups' rb tree linkage sometimes acquires bad entries

nonsense values

i must fix it

oh i see why

it doesn't remove the pgroup from the rb tree

[stone orbit]

tried building keyronex but it fails while building libX11 with following error:

  CC       xim_trans.lo
In file included from /home/czapek/sources/Keyronex/build/system-root/usr/include/X11/Xtrans/Xtranssock.c:110,
                 from /home/czapek/sources/Keyronex/build/system-root/usr/include/X11/Xtrans/transport.c:67,
                 from /home/czapek/sources/Keyronex/ports/libX11/src/xlibi18n/xim_trans.c:30:
/home/czapek/sources/Keyronex/build/system-root/usr/include/sys/ioctl.h:9:17: fatal error: asm/ioctls.h: No such file or directory
    9 | #       include <asm/ioctls.h>
      |                 ^~~~~~~~~~~~~~

.c files in /usr/include sure does sound intriguing

[hexed solstice]

Thats odd

That header is installed with the keyronex-kernel package

Also surprised it went as far as libX11 before it failed, I should have expected problems much earlier

[stone orbit]

hmm interesting, im not able to check it again now but i built the kernel manually so maybe its some dependency issue?

[hexed solstice]

yes, i should put a keyronex-kernel dependency in somewhere along the way

the gnu smalltalk build will also probably fail, i'm sure i forgot to include the patch that makes it cross-compileable

did you have any other issues along the road btw?

[stone orbit]

uhh ksh93 was missing

or ksh98, unsure

but i fixed it easily

[hexed solstice]

you've had far fewer than some guy on #osdev irc

i think i only use ksh93 for the script to launch qemu

[stone orbit]

that is correct, any reason for that?

might be wise to make it a posix shell script

[hexed solstice]

unfortunately i don't know what's posix and what's not

[stone orbit]

eh as long as you do a basic script it should be fine :^)

[zenith karma]

the GUI extensions are totally posix :^)

[hexed solstice]

ksh93 is problematic because i actually can't build it for keyronex yet as it intimately depends on compiling and running test programs

and if i remember right it stressed mlibc with missing localisation stuff

[zenith karma]

what if you build it /on/ keyronex

[hexed solstice]

something i hope to eventually be able to do

i did experiment with Red Hat's fork of ksh93, ksh2020

but they stripped out ksh93's implementation of the fts.h header and functionality, since they ditched most ports of ksh93

[stone orbit]

@hexed solstice the include error is gone now that i've built the kernel manually but instead i get this one:

  CC       x11_xcb.lo
/home/czapek/sources/Keyronex/ports/libX11/src/XlibInt.c: In function 'SocketBytesReadable':
/home/czapek/sources/Keyronex/ports/libX11/src/XlibInt.c:1267:5: error: implicit declaration of function 'ioctl' [-Werror=implicit-function-declaration]
 1267 |     ioctl(ConnectionNumber(dpy), FIONREAD, &bytes);
      |     ^~~~~
/home/czapek/sources/Keyronex/ports/libX11/src/XlibInt.c:1267:5: warning: nested extern declaration of 'ioctl' [-Wnested-externs]
/home/czapek/sources/Keyronex/ports/libX11/src/XlibInt.c:1267:34: error: 'FIONREAD' undeclared (first use in this function)
 1267 |     ioctl(ConnectionNumber(dpy), FIONREAD, &bytes);
      |                                  ^~~~~~~~

[hexed solstice]

oh, that's bizarre, it should be including <sys/ioctl.h> which should define those

oh i think i see what happened

[stone orbit]

yeah i can confirm that sys/ioctl.h does indeed define FIONREAD

maybe some patch you haven't upstreamed?

[hexed solstice]

because <sys/ioctl.h> wouldn't have been working at the time of configure, it probably didn't define some config.h definition to tell it to actually include that file

so hopefully it works with a reconfigure

[stone orbit]

oh nice, reconfiguring worked

hmm now dying when regenerating python

xbstrap: regenerate python [38/232]
xbstrap: Running ['autoreconf', '-fvi'] (tools: ['host-autoconf-2.69'])
autoreconf: Entering directory `.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal --force
autoreconf: configure.ac: tracing
autoreconf: configure.ac: not using Libtool
autoreconf: running: /home/czapek/sources/Keyronex/build/tools/host-autoconf-2.69/bin/autoconf --force
configure.ac:18: error: possibly undefined macro: AC_MSG_ERROR
      If this token and others are legitimate, please use m4_pattern_allow.
      See the Autoconf documentation.
autoreconf: /home/czapek/sources/Keyronex/build/tools/host-autoconf-2.69/bin/autoconf failed with exit status: 1
xbstrap: Action regenerate of source python failed
make: *** [Makefile:9: all] Error 1

[hexed solstice]

that flaming thing

i always seem to see random stuff like that pop up in autoconf with 'possibly undefined macros'

maybe try adding host-pkgconf to the tools_required of python regenerate (userland.yml line no #485)

that's apropos of some random blogpost i found

[stone orbit]

nope, host-pkgconf didnt help

[hexed solstice]

bugger it, i know i have a comment saying otherwise there but try adding also host-automake-1.16 and host-libtool

[stone orbit]

nope :/

[hexed solstice]

and what platform is this? i might try installing it in a VM and see whether i run into issues also

[stone orbit]

im using arch, im almost on the latest version of packages

[hexed solstice]

i reproduced the issue by adding host-libtool, host-pkgconf, etc to the thing

[stone orbit]

hmm interesting

maybe its some host packages that are conflicting

((a major reason why jinx>xbstrap ))

[hexed solstice]

adding host-automake-1.16 but not pkgconf libtool etc gives me even worse results

[stone orbit]

i bet we could make it work on *bsd by using chroot and requiring you to use root

the main reason it doesnt work on !linux is unshare, which we use to emulate a chroot environment with uid/gid 0

it lets you set up uid and gid mappings

[hexed solstice]

the having to build as root i'm not keen on

[stone orbit]

well its only required to chroot into the build root

[hexed solstice]

but that really reflects the fundamental problem that there is no interface available consistently on BSD platforms for non-root sand boxing

[stone orbit]

can you do that as non root?

[hexed solstice]

unfortunately not, chroot is strictly limited to root

i am actually not sure why unshare isn't

[stone orbit]

i wonder if you can just give it setuid permissions so we can chroot and drop the root privileges

[hexed solstice]

there are tricks that can be played with symlinks to gain escalated privileges with chroot and i wonder how unshare protects against those

[stone orbit]

i dont think it does

ive heard that namespaces are actually one of the biggest sources of kernel bugs

[hexed solstice]

i know that namespaces RUINED pidfd

they put stupid checks into the pidfd code like "is the current process that wants to send a signal to the process designated by a pidfd in the same namesapce, is it privileged to send the signal"

the whole point of an FD is that by having it with particular permissions, you are authenticated to use it

[zenith karma]

if you unshare whatever "root" you gain is worthless

its not the real "root" outside the namespace

so you cant do priviledged stuff

(afaik)

[hexed solstice]

you still have root privileges in all other respects

[zenith karma]

like what aspects, exactly?

[hexed solstice]

if it's just the filesystem namespace that's unshared then you are free to send SIGKILL to any process you will

[zenith karma]

of course

but you usually try to unshare all of them

[stone orbit]

are you sure?

im pretty sure unsharing the PID namespace prevents that

[zenith karma]

you could open a pidfd

actually nvm

[hexed solstice]

but unshare is not privileged

[zenith karma]

that wont work

[stone orbit]

it doesnt work across namespaces, yeah

[zenith karma]

it is

[hexed solstice]

oh, that changes everything

[zenith karma]

the only unshare that isn't is user unshare

but user unshare drops all host perms basically afaik

[hexed solstice]

i thought it wasn't and that this is why it's used in jinx

[zenith karma]

jinx does a user unshare first iirc

actually it seems to do a pid unshare at the same time as well

aye

you can only do an unshare if you are root

or well

if you have the cap

and you get the cap if you do a user unshare

or are root

(i think)

[sinful agate]

Not if you run cbuildrt which is recommended!

[zenith karma]

does "recommended" not mean that it is optional

[sinful agate]

We don’t want to force anything

So yes

Just like how it’s highly recommended, but not required, to build managarm with cbuildrt

Tho I’m pretty sure I was the last person that didn’t do that

[hexed solstice]

diff --git a/userland.yml b/userland.yml
index 96b46b0..ce4034a 100644
--- a/userland.yml
+++ b/userland.yml
@@ -482,9 +482,12 @@ packages:
       patch-path-strip: 1
       regenerate:
         # python is broken, adding automake or libtool ruins it
-        - args: [ 'autoreconf', '-fvi' ]
+        - args: [ 'autoconf' ]
       tools_required:
         - host-autoconf-2.69
+        - host-automake-1.16
+        - host-libtool
+        - host-pkgconf
     tools_required:
       - host-gcc
       - host-pkgconf

this might work

it did for me

it probably won't for you because python's autoconf is fuxy-wuxy

[hexed solstice]

i wonder in how many wonderful ways it would fail on netbsd

i also don't think i have any fundamental objection to an suid binary for chroot. i do wonder whether some dirty tricks with LD_PRELOAD might provide for a non-root sandbox

[stone orbit]

ugh what???

xbstrap: Running ['@THIS_SOURCE_DIR@/configure', '--host=x86_64-keyronex', '--build=x86_64-netbsd', '--prefix=/usr', '--enable-shared', '--with-sysroot=@SYSROOT_DIR@', '--with-readline', '--with-system-ffi', '--with-system-expat', '--disable-ipv6', '--without-ensurepip', '--with-build-python=@THIS_BUILD_DIR@/python3.11'] (tools: ['host-pkgconf', 'host-gcc', 'host-binutils'])
configure: WARNING: unrecognized options: --with-sysroot
configure: loading site script /home/czapek/sources/Keyronex/build/pkg-builds/python/confsite
configure: error: Please install autoconf-archive package and re-run autoreconf
xbstrap: Action configure of package python failed

[hexed solstice]

i suppose it must be wanting host-autoconf-2.69 in tools_required for the build as well as regen

what an annoying pos

[tiny sapphire]

libtool memes?

[stone orbit]

adding autoconf:

configure: WARNING: unrecognized options: --with-sysroot
configure: loading site script /home/czapek/sources/Keyronex/build/pkg-builds/python/confsite
configure: error: Please install autoconf-archive package and re-run autoreconf
xbstrap: Action configure of package python failed

[hexed solstice]

it wants AX_C_FLOAT_WORDS_BIGENDIAN to be defined apparently

[sinful agate]

Host-autoconf-archive for the win

[hexed solstice]

yeah, it sounds like it needs one like that with export_aclocal appropriately set

i wonder why it didn't affect me

presumably it used my host autoconf at some stage

you might snatch host-autoconf-archive from https://github.com/managarm/bootstrap-managarm/blob/0e9f9c33fddd585565da023536f6c4dcf1f6b3e5/bootstrap.yml and add it as a required tool on both stages

thank you btw for trying to build this thing

you are the first, i think, apart from some guy on #osdev irc who had even more severe problems

[hexed solstice]

i cannot persuade pacstrap to work

the fucker deleted all the packages it downloaded after spitting repeated errors like "error: base: signature from "Jonas Witschel [email protected]" is unknown trust"

not happy with it having to re-download everything every time i try and fail to make it work - i wonder if i can just shut off this key verification nonsense

[quick saddle]

I'm not sure if pacstrap follows /etc/pacman.conf but try changing the SigLevel for the required repositories

https://man.archlinux.org/man/pacman.conf.5#PACKAGE_AND_DATABASE_SIGNATURE_CHECKING

[hexed solstice]

cheers, that appears to have done the needful

[zenith karma]

you could try using a caching http proxy

[stone orbit]

💀

@@ -508,8 +526,7 @@ packages:
       - args: ['chmod', '+x', '@THIS_BUILD_DIR@/python3.11']
       - args:
         - '@THIS_SOURCE_DIR@/configure'
-        - '--host=x86_64-keyronex'
-        - '--build=x86_64-netbsd'
+        - '--target=x86_64-keyronex'
         - '--prefix=/usr'
         - '--enable-shared'
         - '--with-sysroot=@SYSROOT_DIR@'
@@ -518,7 +535,6 @@ packages:
         - '--with-system-expat'
         - '--disable-ipv6'
         - '--without-ensurepip'
-        - '--with-build-python=@THIS_BUILD_DIR@/python3.11'
         environ:
           CONFIG_SITE: '@THIS_BUILD_DIR@/confsite'
           PKG_CONFIG_SYSROOT_DIR: '@BUILD_ROOT@/system-root'

that makes it build

of course this assumes your host has python 3.11

you should build host-python to make it happy

checking for SHA1Init... no
checking for CC_SHA1_Init... no
checking for wincrypt.h... no
checking for SHA1Init in -lmd... no
checking for LIBSHA1... no
checking for nettle_sha1_init in -lnettle... no
checking for gcry_md_open in -lgcrypt... no
checking for SHA1_Init in -lcrypto... no
checking for OPENSSL... no
checking for SHA1 implementation... configure: error: No suitable SHA1 implementation found
xbstrap: Action configure of package xorg-server failed
``` fun

apparently its missing gcrypt? or something

oh it also can take nettle!

i think thats what lyre uses

[hexed solstice]

i was using openssl

must've neglected to add it to the xorg-server deps

[stone orbit]

oh ok

ill add it then

[hexed solstice]

did you end up needing the autoconf-archives stuff btw?

or did it build python without?

[stone orbit]

uhh not sure, but i added it just in case

configure: error: Please install InfoZIP to use GNU Smalltalk.
xbstrap: Action configure-tool of tool host-smalltalk failed

i got this while building smalltalk

[hexed solstice]

oh, i expect that recipe to fail anyway

[stone orbit]

is that something expected?

[hexed solstice]

i'd comment it out for now as it's not requierd by anything

[stone orbit]

alright sweet

[hexed solstice]

i think i had to make several hacks to get it to cross-compile and i don't think i put them into the recipes/patches yet

[stone orbit]

ill deliver a list of packages that i needed to build keyronex after i build everything

and ill clean up some recipes and add host-python

[hexed solstice]

cheers for that, this will be a good step forward

i can set up some github actions to build it automatically and i'll get around to having that generating a distribution of the sysroot, iso, and a qemu running script that doesn't depend on ksh93

[stone orbit]

ill make sure it also builds on something generic like debian

that would be perfect in that case

[hexed solstice]

yeah, i'm hopeful to get it to a state where it's easily buildable without having to hack your way through the process

till now i only built it under fedora and netbsd and that in an ad-hoc way, not from scratch

[stone orbit]

eh it wasn't too bad, except for the libX11 part where kernel headers weren't installed 😁

[tiny sapphire]

reminds me of aero buildsystem

it sucks

cocks

[hexed solstice]

yeah, i'm trying to think of where to insert a dependency on the keyronex kernel

actually

[tiny sapphire]

but it works on my machine (tm)

[hexed solstice]

i should maybe add a headers-only target to it and have mlibc package depend on it

[stone orbit]

xclock has that same autoconf issue as python, interesting

[stone orbit]

that is a good idea actually

[hexed solstice]

i think i had an issue with AM_ICONV which i deleted to good effect

autoconf would be a lot easier to grapple with if there weren't a million different variations of how it's used which seems to lead to these problems

[hexed solstice]

that you got llvm and rust to build at all shows determination

those two frighten me

[tiny sapphire]

they are scary indeed

except rustc for me, maybe cauz i know rust

[hexed solstice]

i'd like llvm in the future for mesa's fast software renderer and for clang's better support of objc

[tiny sapphire]

i want to upstream llvm so i can upstream rustc

[hexed solstice]

on the other hand arsen is urging me to join the GCC project to improve support there

[tiny sapphire]

but idk if they would accept

does llvm accept projects like aero?

[stone orbit]

apparently missing AM_ICONV can be solved by installing gettext but i already have that...

[hexed solstice]

it does build with AM_ICONV commented out, which is what i think i did

why it doesn't pick up gettext i don't know

[stone orbit]

oh lmao

[hexed solstice]

nothing under tools defines AM_ICONV i notice which surprises me

[stone orbit]

hmm alright, everything is built now!

[hexed solstice]

oh, i don't even build gettext, i thought i did

[stone orbit]

im asserting in start.c:60

    r = vfs_lookup(root_vnode, &dev_shadowed, "dev", 0, NULL);
    kassert(r == 0);

[hexed solstice]

oh i see

mkdir build/system-root/dev

[stone orbit]

yeah it worked, boots into bash!

it took it a while but it worked 😁

[hexed solstice]

shit, the first non-me boot of keyronex

[stone orbit]

ls didnt work though :^(

it panicked in readdir

in mlibc of course, sorry

[hexed solstice]

oh dear

what did it complain about?

oh

that's a surprise

[stone orbit]

v9fs_read tag 638 id 116 fid 22 off 528384 max_count 4096
v9fs_read_return tag 638 id 116 count 4096 err 4107
v9fs_read tag 639 id 116 fid 22 off 696320 max_count 4096
v9fs_read_return tag 639 id 116 count 4096 err 4107
v9fs_walk tag 640 id 110 fid 1 newfid 66 nwnames 0
v9fs_walk_return tag 640 id 110 nwnames 0 qids (nil)
v9fs_open tag 641 id 12 fid 66 mode 65536
v9fs_open_return tag 641 id 12 qid={type 128 version 1683901827 path 4497449} iounit 0
v9fs_readdir tag 642 id 40 fid 66 offset 0 max_count 2048
v9fs_readdir_return tag 642 id 40 count 215 retval 226
<DEBUG>: In function readdir, file ../../../mlibc/options/posix/generic/dirent-stubs.cpp:72
__ensure(!"mlibc::sys_read_entries() failed") failed
<DEBUG>: 
MLIBC PANIC

[hexed solstice]

it looks like 9p worked as expected, bizarre

[stone orbit]

well it clearly returned an error :p

also i noticed a small issue in the sysdeps:```
uintptr_t r = syscall3(kPXSysReadDir, fd, (uintptr_t)buffer, max_size,
NULL);

shouldnt it be syscall4?

[hexed solstice]

oh, there's an optional out parameter in each of these

[stone orbit]

ohh that makes sense

[hexed solstice]

i don't think i use it at all right now

i wonder what error it is

kernel/posix/sys.c before line 420 insert kprintf("error %d\n", r);

[stone orbit]

hmm 🤔 ```
at ../../../kernel/driver/viofam/9pfs_reg.c:116 (ninep_buf_addstr):
kernel assertion failed: newoff <= buf->bufsize + sizeof(struct ninep_hdr)

[hexed solstice]

that's a stinker

[stone orbit]

nothing

hold on a second, ill make sure i recompiled

oh i have to rebuild-kernel

[hexed solstice]

i don't like the ninepbuf code, i must try and figure out why that happened

[stone orbit]

jesus

[hexed solstice]

yeah, the default target just does xbstarp install --all

[stone orbit]

error -1

EPERM

[hexed solstice]

that must be coming from the 9p side as i don't think i generate EPERM at all yet except for process group stuff

[stone orbit]

well isn't that wonderful

ugh

[hexed solstice]

yet

v9fs_readdir_return tag 642 id 40 count 215 retval 226
this is bizarre

[stone orbit]

what even is that?

i cant find any logs regarding that anywhere

[hexed solstice]

it's from qemu itself

[stone orbit]

oh what the hell?

[hexed solstice]

your qemu (unlike mine) appears to actually respect the trace options

[stone orbit]

interesting

sooo... is qemu emulating some sort of a filesystem for you?

or how does that work

[hexed solstice]

it's got a built in 9p server

[stone orbit]

ohh i see```
(($virtio9p)) && {
qemu_args+="-virtfs local,path=${root},security_model=none,
mount_tag=root "
}

interesting

[hexed solstice]

in driver/viofam/9pfs.cc it might be fun to add a kprintf("9p readdir lerror %d", err); before goto out;

and if that doesn't come up with an error then something really weird is going on

[stone orbit]

is rebuild-kernel enough here too?

[hexed solstice]

yeah, that will do it

though qemu logs suggest no lerror is returned which is very bizarre

[stone orbit]

yeah, no error!

i even added a check right before return r

if r < 0 kprintf

[hexed solstice]

by god, i think i might have it

i think maybe qemu v9fs returns -1 to indicate that there are no more entries to readdir

[stone orbit]

💀

[hexed solstice]

and then on mlibc end that mistakes it for a real error

[stone orbit]

that would kinda make sense...

[hexed solstice]

yet i never ran into this myself, maybe different qemu versions adhere more closely to 9p spec

well, 9p doesn't have a spec, which is the problem

[stone orbit]

after adding if (r == -1) return 0; before the error check i can now list /dev

but that is not the solution

because i cant list anything else hehe

[hexed solstice]

yeah, it looks like on my qemu 9p readdir returns not -1 for no more available to read

it gives e.g. 512 and then the next attempt to readdir gets the same last offset but zero entries

[stone orbit]

not -1 as in ~(-1)? or a different value

oh alright, makes sense

hmm i wonder how do different kernels handle the case when the contents of a directory change in between readdir calls? is the user supposed to keep track of that? does the kernel take sort of a "snapshot"? it's always bugged me

[hexed solstice]

i will prepare a patch shortly for this

[hexed solstice]

the general approach i have seen is: no snapshot is taken, the kernel is careful with the offset values

so e.g. someone tries to get dirents at offset 720 on a filesystem with 512-byte blocks

[stone orbit]

so basically if someone messes with the directory the user is iterating over then thats their problem?

[hexed solstice]

the fs driver goes back to offset 512 into the directory (because offset 720 may no longer be meaningful) and it then iterates until it gets to first dirent that it should start returning based on the offset value given

[stone orbit]

i wasn't really thinking from the kernel's perpective, rather the user's perspective

[hexed solstice]

posix specifies something or other to do with this

actually i am now a bit surprised at how -1 got to userland, as sys_readdir returns number of bytes read and not the offset to begin reading at again

[stone orbit]

i feel like it might be some faulty cast along the way

9p readdir result 9223372036854775807

lovely

[hexed solstice]

what the fuck

[stone orbit]

:D

i think i might've fucked up

how do you exactly print a long?

i used %ld

[hexed solstice]

that should be it

[stone orbit]

well then, there it is

[vale pelican]

0x7fffffffffffffff hmmmm

[hexed solstice]

if i throw in e.g. kprintf("last_offset = %ld, r = %ld\n", last_offset, r); to NinePFS::readdir() just before out: i get a rather pleasant last_offset = 512, r = 512 for ls /

[stone orbit]

uh oh

i get last_offset = 9223372036854775807, r = 9223372036854775807

[hexed solstice]

how about bytes_copied_out?

[stone orbit]

192

[hexed solstice]

that sounds more like it

let's see if the format of 9p readdir is different between our qemus

[stone orbit]

im on 7.2.1

[hexed solstice]

7.0.0 here

[stone orbit]

and i have an update pending

update to 8.0.0, woo

im not going to just yet

then we can see the difference between 7.2.1 and 8.0.0

[hexed solstice]

before kmem_strfree(name) in the ninepfs readdir(), add kprintf("Off = %lu, ino = %lu, type = %d, name = %s\n", offset, qid.path, type, name);

meanwhile i will build qemu 8.0.0

[stone orbit]

oh thats nasty

[hexed solstice]

yeah, very bad

[stone orbit]

middle finger to qemu

why do they have to do this

[hexed solstice]

what's your host fs? i've noticed qemu's 9p implementation leaks some details

for example, if those inode numbers match the results of stat on your host fs, i wouldn't be surprised

[stone orbit]

ext4

[hexed solstice]

ext4 i wouldn't expect any grief from

[stone orbit]

hahahah

[czapek@DESKTOP-9SB6964 Keyronex]$ stat build/system-root/* | grep -i ino
Device: 8,32    Inode: 4546640     Links: 2
Device: 8,32    Inode: 4613482     Links: 2
Device: 8,32    Inode: 4497470     Links: 5
Device: 8,32    Inode: 4613467     Links: 3
Device: 8,32    Inode: 4497450     Links: 7
Device: 8,32    Inode: 4571459     Links: 5

you wouldnt

why the hell does it do that??

jesus

also any idea what's up with the offset field?

[zenith karma]

how else would it get imode numbers!

there is clearly no other way!

[final hatch]

imode

[stone orbit]

generate them, come on :(

[hexed solstice]

oh qemu 8.0.0's trace does give me the v9fs stuff

[zenith karma]

what does generation mean?

i mean clearly worse than just giving you the host ones

[stone orbit]

yeahhh you're right :/

[hexed solstice]

i must figure out what's going on with these offset values

this is the downside of booting from a somewhat leaky abstraction i suppose

going to make an ext4 image and see if i can get similar nonsense from qemu

yes i do

[stone orbit]

interesting, im reading various stuff i found on google and i see that the readdir response is { u64 ino; u64 offset; u8 type; u16 name_len; char name[name_len]; }

[hexed solstice]

and i am treating it as an off_t (which is 64-bit, but signed)

[stone orbit]

interesting, shouldn't matter though

i mean the inode is correct

so that stuff i found already seems wrong...

[hexed solstice]

i can now reproduce your issue at least

[stone orbit]

with qemu 8?

neat

[hexed solstice]

yeah, but only with an ext4 drive backing system-root

[stone orbit]

does it work with whatever else you were using earlier?

[hexed solstice]

yes, fine on ZFS on NetBSD, XFS on Fedora

i'm going to throw some printfs into qemu and determine where it's getting the offset from and how

[stone orbit]

hmm so it might be an ext-only issue after all?

ok yeah ok that is a great idea

[hexed solstice]

it seems so, but i have't heard of others having the issue, so i am hoping to fix it

[zenith karma]

qemu is on version 8 now?

[stone orbit]

yeah, apparently

[hexed solstice]

there we go

(don't mind the other junk in my system root)

[zenith karma]

what are these offsets as hex?

[hexed solstice]

well, 9223372036854775807 is INT64_MAX which is very interesting to begin with

[stone orbit]

for me

[final hatch]

could that signify the end of the directory listing?

[zenith karma]

just random crap it seems

[hexed solstice]

it seems it must do

[stone orbit]

the -1 on "." seems weird

it should at least point to tiself, somehow

[hexed solstice]

it seems meaningful to me that they form an ordered sequence

as if they are synthetic values, or maybe represent something funny like keys in some kind of tree

[stone orbit]

also which part of keyronex is written in objc?

i see objfw in the project root but i didnt really see any objc code...

all i see is c and c++

[hexed solstice]

the previous driver framework

[stone orbit]

ohh, was everything rewritten to c++?

[hexed solstice]

yeah, i had a very enthusiastic person ask to contribute but refused to touch objc

so i translated it to c++ like a rust crab on a bad trip

[stone orbit]

lmao

[hexed solstice]

then he fucked off after my hard work was done

[stone orbit]

oh :( sorry

i might contribute then, its a very nice project

but im unsure which parts actually need work

i feel like an ext2 driver would be nice

[hexed solstice]

cheers, there are unfortunately numerous hacks all over the place right now and a lot of parts not currently working

[stone orbit]

nothing that can't be fixed :p

[hexed solstice]

for example the initial mounting of things is a mess of hand-written setting of struct members; vfs_lookup() somehow works most of the time, but it was apparently messing up refcounts on vnodes so vnodes don't get refcounted currently

[hexed solstice]

aye, there is a lot of work to do to whip it into shape, but i think that it can run xorg shows that it's at least not truly irredeemable

on the other hand it would probably not take well to an X client exiting as unix sockets are implemented in the most minimal fashion to get X to display xeyes and run twm

if you felt brave enough to tackle this thing despite the problems and the arcaneness of a lot of the source base i would welcome it

[stone orbit]

id definitely like to contribute, i cant get myself to write something nice of my own so i think that's the second best option

im sure i can get around the codebase after i spend some time with it, the code seems well written and i've seen some comments around the place so it cant be that bad

[hexed solstice]

i have a fix for our problem btw

diff --git a/kernel/posix/sys.c b/kernel/posix/sys.c
index ee11b79..47cc755 100644
--- a/kernel/posix/sys.c
+++ b/kernel/posix/sys.c
@@ -401,7 +401,7 @@ sys_readdir(int fd, void *buf, size_t bufsize)
 {
        struct file *file = ps_getfile(ps_curproc(), fd);
        size_t bytes_read = 0;
-       int r;
+       off_t r;
 
 #if DEBUG_SYSCALLS == 1
        kdprintf("SYS_READDIR(fd: %d, bufsize: %lu off: %lu)\n", fd, bufsize,

[stone orbit]

oh wow

[hexed solstice]

that's the damned issue

[stone orbit]

wait really?

what about the weird offsets?

oh wow it does work

[hexed solstice]

i don't know why i put an int there when it returns off_t but that's apparently the issue. it seems the weird offsets are just what qemu's 9pfs server in conjunction with underlying ext4 generates, and i can confirm it works even on ls /usr/bin which needs multiple readdir() calls

[stone orbit]

yeah it does work, that is so weird

good catch!

soo, how do i run xorg? do i just X &; twm &; xeyes &; xclock &?

oh xorg crashed :(

[hexed solstice]

X&, wait for it to go, export DISPLAY=:0, xeyes&/twm&/etc

bollocks, what happened with it?

[stone orbit]

backtrace in mlibc not implemented

<DEBUG>: In function backtrace, file ../../../mlibc/options/glibc/generic/execinfo.cpp:5
__ensure(!"Not implemented") failed

[hexed solstice]

oh yeah

[stone orbit]

did you stub that out?

[hexed solstice]

i keep forgetting to upstream the patch out of that

or rather commit it to the repo

i got rid of a call to backtrace() in xorg-server somewhere, one sec

[stone orbit]

that's one way to solve it 😄

[hexed solstice]

or did i, hmm, i can't find it

[vale pelican]

for backtrace perhaps you could port libbacktrace?

it's what musl systems use in place of backtrace from glibc afaik

[hexed solstice]

lmao i literally just deleted the code, didn't even comment it out, that's why i couldn't find it

linse 209 to 219 of ports/xorg-server/os/osinit.c

[hexed solstice]

that seems reasonable, i notice in the objfw port i get backtraces printed out, it doesn't use backtrace() of course

[stone orbit]

got rid of it, recompiling now

[hexed solstice]

there may also be an epoll issue in xorg with multi threads but i'll wait and see if that turns out to be a problem for you

[stone orbit]

interesting

[hexed solstice]

objfw appears to be using _Unwind_Backtrace()

[vale pelican]

ah yeah that's from libgcc or one of the adjacent libraries

[hexed solstice]

this is one of the more embarassing ones, i haven't implemented mkdir in 9pfs yet so you've got to mkdir build/system-sysroot/tmp and build/system-root/mtp/.X11-unix on the host

(you can tell i was in a hurry to get xeyes looking at me)

[stone orbit]

v9fs_walk tag 1599 id 110 fid 1 newfid 142 nwnames 1
v9fs_rerror tag 1599 id 110 err 2
v9fs_walk tag 1600 id 110 fid 1 newfid 143 nwnames 1
v9fs_rerror tag 1600 id 110 err 2
v9fs_walk tag 1601 id 110 fid 1 newfid 144 nwnames 1
v9fs_rerror tag 1601 id 110 err 2
v9fs_walk tag 1602 id 110 fid 1 newfid 145 nwnames 1
v9fs_rerror tag 1602 id 110 err 2
v9fs_walk tag 1603 id 110 fid 1 newfid 146 nwnames 1
v9fs_rerror tag 1603 id 110 err 2
v9fs_walk tag 1604 id 110 fid 1 newfid 147 nwnames 1
v9fs_rerror tag 1604 id 110 err 2
v9fs_walk tag 1605 id 110 fid 1 newfid 148 nwnames 1
v9fs_rerror tag 1605 id 110 err 2
v9fs_walk tag 1606 id 110 fid 1 newfid 149 nwnames 1
v9fs_rerror tag 1606 id 110 err 2
v9fs_walk tag 1607 id 110 fid 1 newfid 150 nwnames 1
v9fs_rerror tag 1607 id 110 err 2
v9fs_walk tag 1608 id 110 fid 1 newfid 151 nwnames 1
v9fs_rerror tag 1608 id 110 err 2
v9fs_walk tag 1609 id 110 fid 1 newfid 152 nwnames 1
v9fs_rerror tag 1609 id 110 err 2
v9fs_walk tag 1610 id 110 fid 1 newfid 153 nwnames 1
v9fs_rerror tag 1610 id 110 err 2
v9fs_walk tag 1611 id 110 fid 1 newfid 154 nwnames 1
v9fs_rerror tag 1611 id 110 err 2
v9fs_read tag 1612 id 116 fid 22 off 368640 max_count 4096
v9fs_read_return tag 1612 id 116 count 4096 err 4107

ohh

that makes sense lmao

ill see if i can do that right now

seems like a nice first thing to do

[vale pelican]

hmm i suppose we could implement backtrace in mlibc using _Unwind_Backtrace

[hexed solstice]

also sys_rename() is absent (xorg uses that specifically rather than link and unlink, iirc) and this means that i always prepend rm -f build/system-root/var/log/Xorg.0.log && rm -f build/system-root/tmp/.X0-lock to clear old files

[vale pelican]

although that doesn't seem to provide the ip->symbol name functionality that backtrace provides

[stone orbit]

another missing directory? :p

[hexed solstice]

i think it might be lmao

[hexed solstice]

here is what objfw does: https://github.com/ObjFW/ObjFW/blob/d436e8f4661bd2359bd90529c544088a19f1b2f6/src/exceptions/OFException.m

[vale pelican]

oh it just uses dladdr

[hexed solstice]

see if Xorg.0.log has any info, i can't remember which directory

libbacktrace i imagine prints a nicer backtrace with line-numbers and suchlike but dladdr could be worse

[stone orbit]

oh here it is

[     0.000] (EE) XKB: Couldn't open rules file /home/czapek/sources/Keyronex/build/system-root/usr/share/X11/xkb/rules/base
[     0.000] (EE) XKB: Failed to load keymap. Loading default keymap instead.
[     0.000] (EE) XKB: Couldn't open rules file /home/czapek/sources/Keyronex/build/system-root/usr/share/X11/xkb/rules/base

[vale pelican]

iirc it doesn't

[hexed solstice]

oh, that's interesting, i do have that directory and that /base file and i could've sworn the files in it are generated in the build process

[stone orbit]

let me see if a rebuild of xkbmap will do anything

[hexed solstice]

yeah, i think i might have forgotten to specify dependencies properly, maybe on libxml2

[stone orbit]

libxkbfile and xkbcomp

oh there it is now

interesting

[hexed solstice]

xeyes btw will probably not display anything because i had to edit its generated config.h to remove #define XPRESENT 1 (i don't know why this doesn't work on keyronex)

[stone orbit]

uhh still nothing, the same error

lets check the log out

OH WOW

Couldn't open rules file /home/czapek/sources/Keyronex/build/system-root/usr/share/X11/xkb/rules/base

it uses a host path!!!

no wonder it cant open it

[hexed solstice]

i think i had this exact issue actually

let me try and remember what i did to deal with it

i think these lines might fix it in the configure args for the xorg-server recipe:
--with-xkb-bin-directory=/usr/bin
--with-xkb-path=/usr/share/X11/xkb

this problem came and went for me and i don't know why

[stone orbit]

kfatal("Failuer!\n");

in exec

[hexed solstice]

lmao

it's trying to exec something nonexistent

[stone orbit]

oh right

/bin/sh

[hexed solstice]

i can't remember why i put a kfatal there

[stone orbit]

missing link from /bin -> /usr/bin?

[hexed solstice]

yeah, and possibly a missing sh->bash symlink

[stone orbit]

that one is actually there

yaaay!

it did it

sadly i cant place down the xeyes window, but it did show one!

and xclock is just a tiny little pill in the top left

also ls -l hangs the system on a sys_open :^)

[hexed solstice]

fantastic

[hexed solstice]

i think it might be trying to print a string in demand-paged area

this will page fault which will complain about the IPL being too high but fail to acquire the dprintf spinlock

[stone orbit]

oh sweet

[hexed solstice]

that's usually what's behind these errors with half-printed text

[stone orbit]

that's a first for me :^)

so what is the way to fix this?

like, properly i guess

[hexed solstice]

it's helpful to know as it means that i am accessing userspace memory directly

[stone orbit]

is that bad?

[hexed solstice]

it might upset some filesystems which acquire spinlocks (page faulting with a spinlock held is illegal) before working with the path, and it's also just outright wrong since userspace data shouldn't be trusted like that

so one thing i have on the agenda is a set of functions for copying data from userland safely

in this case e.g. user_strdup() or something like that to try to copy a string from userland memory safely

and then work with that copy instead

for now a simple strdup is fine and will do the job

[stone orbit]

it seems like this is exactly what is being done inside vfs_lookup

it copies from path to a local buffer char name[255]

wouldn't it page fault there instead? and if so why is that any better?

[hexed solstice]

it's not good but it wouldn't crash (as long as the address is valid) as it's not holding any spinlocks

while with that printf of sys_open(....) it crashes since it's in the middle of printf with the dprintf spinlock held

[stone orbit]

OH

it's all about the print lock!

oh also, ive been meaning to ask one thing about interrupt levels

if you're in a high ipl that doesn't allow certain class of interrupts to interrupt you, what happens after ipl is dropped? are the interrupts queued somewhere and handling is retried automatically?

[hexed solstice]

for hardware interrupts we take advantage of the cr8 register

this is associated with the APIC

APIC masks interrupts whose vector is < the value of cr8 << 4

so when the ipl is dropped later, the interrupts that were previously masked get delivered

[stone orbit]

so the apic automagically delivers the interrupts?

[hexed solstice]

DPCs on the other hand get put onto a queue and when IPL drops below DPC level, that queue is processed. i could send a self-IPI instead with the appropriately low vector to do this if i really wanted to

[hexed solstice]

yeah, for ports to other platforms without similar, if i want hardware interrupt prioritisation i would need to implement a software based mechanism instead

[stone orbit]

now, doesnt that make it a hell to route the interrupts in a certain way so that the vectors are ordered properly for different kinds of interrupts?

[hexed solstice]

the situation isn't too bad, it just means that for a particular IPL, i have 16 vectors i can use for that IPL

so it's just iterate from IPL << 4 to (IPL << 4) + 15 to look for a free slot

and in many cases vectors can be shared

[stone orbit]

how does the device ipl get determined?

[hexed solstice]

i think i keep some high number which i can't remember for things like the tlb shootdown ipi

and then for the rest, right now i think i am using arbitrary numbers because i haven't had an actual need for hardware interrupt prioritisation yet

[stone orbit]

i see, thats interesting

[sinful agate]

Setup xbbs

[sinful agate]

Llvm ain’t too hard

[stone orbit]

that isnt free :^)

you need a server to do the building for you

github actions is as free as it gets imho

or maybe oracle cloud, but that is arm

[zenith karma]

what if xbbs

but with gha as workers

[sinful agate]

Join managarm plz

[zenith karma]

you make a dummy repo

put in 16 branches

[stone orbit]

i am not coming anywhere near that kernel

[sinful agate]

Again. Comment out the check in meson.build. It’s what we do

[zenith karma]

then you write the ID of the job and some manifest stubs into the repo

and force push to a ref

[sinful agate]

Honestly the coordinator from managarm has room to spare and I have a worker available. Ez clap

[sinful agate]

Sadge

[stone orbit]

What's included with Oracle Cloud Free Tier?
Up to 4 instances of ARM Ampere A1 Compute with 3,000 OCPU hours and 18,000 GB hours per month

i have no idea what an OCPU or GB hour is

[zenith karma]

OCPU is probably 1 cpu core

[stone orbit]

yeah

[zenith karma]

you have 18000 hours of 1GB

or 9000 hours of 2GB

etc

or 24 GB a month

[stone orbit]

according to that 31*24*4 comes in at just under 3000

[zenith karma]

bingo

[stone orbit]

seems like a nice place to set up xbbs

:^)

4 A1 cores and 24GB of ram is plenty tbh

wow i cant believe they let me verify for cloud services with a single use card

usually those dont go through

ugh it didnt go through fully

[zenith karma]

lol

[stone orbit]

Error processing transaction

We're unable to complete your sign up. Common sign up errors are due to: (a) Using prepaid cards. Oracle only accepts credit card and debit cards (b) Intentionally or unintentionally masking one's location or identity (c) Entering incomplete or inaccurate account details. Please try again if this applies to you. Otherwise, contact Oracle Customer Service.

???

i wonder if i have to spell out my city name properly

the verification went through

Forbidden

The number of requests has been exceeded. Reload the page or retry the operation. If the problem persists, then contact Oracle Support

im done ¯_(ツ)_/¯

it didnt go through with 2 of my bank cards either, piece of shit

[hexed solstice]

what did these turn out to be?

i've just been adjusting the xbstrap definitions in line with your experience and hopefully a fresh build in future will Just Work

also defined rename and mkdir vnode ops (and implementing these in 9pfs, since i'm using the 9p2000.l variant, couldn't be easier because it has direct counterparts)

[stone orbit]

i had to additionally install python 3.11 + pip, xbstrap, meson and help2man on the host

and i had to modify a few recipes, i should upstream that along with host-python so its easier to build on non rolling distros

[hexed solstice]

that would be handy, i was just looking at your zigux btw, it looks decent - what killed it?

[stone orbit]

hmm, i'd say lack of motivation, the amount of changes made to the zig compiler, it had some bugs and miscompilations so i'd say the state of the tooling for zig (even though it is amazing given how good the language is) and some bad design choices i made

@hexed solstice im not sure if that's the right way to fix it :p https://github.com/Keyronex/Keyronex/commit/a2d959eb49d548cd9e3c39558d1b487f822b9edf#diff-584084b1e813c4481d0c74d3da5d3748b5cf31f2d8437bb0da6e9aa5b5f0b3abR375

honestly im thinking it would be better to just build keyronex-kernel first in the all rule

[hexed solstice]

i suppose i could actually have mlibc-headers outright depend on a keyronex-kernel-headers target

which itself depends on nothing

[stone orbit]

diff --git a/Makefile b/Makefile
index e1d2278..816c2bf 100644
--- a/Makefile
+++ b/Makefile
@@ -6,7 +6,7 @@ ISO_DIR=build/isoroot
 ISO=build/barebones.iso
 
 all: build/
-       (cd build && xbstrap install -c --all)
+       (cd build && xbstrap install -c keyronex-kernel && xbstrap install -c --all)

[stone orbit]

yeah that sounds even better

but i figured because you did it the way you did it you havent gotten around making a kernel-headers target yet :p

[hexed solstice]

just on my way to do that

somehow i have made a circular dep and xbstrap irritatingly doesn't tell me what

i suppose at least it actually prints 'circular dependencies' instead of inscrutable stack traces like it does for other problems

[stone orbit]

yeah xbstrap is a piece of work...

also one more patch for you:```
@@ -521,7 +521,6 @@ packages:
- '--with-system-expat'
- '--disable-ipv6'
- '--without-ensurepip'

•    - '--with-build-python=@THIS_BUILD_DIR@/python3.11'
     environ:
       CONFIG_SITE: '@THIS_BUILD_DIR@/confsite'
       PKG_CONFIG_SYSROOT_DIR: '@BUILD_ROOT@/system-root'
  

ill try building on debian in a second when its done building

[hexed solstice]

i am defeated

[stone orbit]

can you show your diff?

[hexed solstice]

i can't figure out this alleged circular dep

[stone orbit]

im assuming its something you just changed so maybe i could take a look and try helping

[hexed solstice]

it's doubtless just something obvious i am missing through fatigue https://gist.github.com/netbsduser/2ad6b5e496007d1d13ed19ed8b89463b

[stone orbit]

ill take a look in just a second, just going to leave this here - an error configuring xclock:```
checking for XCLOCK... no
configure: error: Package requirements (xaw7 xmu x11 xproto >= 7.0.17) were not met:

Package 'xaw7', required by 'virtual:world', not found

Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.

Alternatively, you may set the environment variables XCLOCK_CFLAGS
and XCLOCK_LIBS to avoid the need to call pkg-config.
See the pkg-config man page for more details.
xbstrap: Action configure of package xclock failed

[hexed solstice]

ah, that will be missing libXaw in pkgs_requierd

[stone orbit]

ah makes sense

[hexed solstice]

i hope

[stone orbit]

can you try adding implict_package: true to keyronex-kernel-headers?

though i dont actually think it should be required

ugh hold on

[hexed solstice]

xbstrap: Failed to validate boostrap.yml
xbstrap: * Error in file: ../bootstrap.yml, YAML element: packages/2
Additional properties are not allowed ('implicit_package' was unexpected)

[stone orbit]

implict

[hexed solstice]

boostrap.yml

[stone orbit]

its a typo in xbstrap :^)

[hexed solstice]

implicit is also disallowed apparently

[stone orbit]

you typed implicit

lose the i, someone made a typo when implementing that :p

[hexed solstice]

magic, that fixed it, cheers

[stone orbit]

oh wow it did? amazing

[hexed solstice]

i don't even know what an implicit package is meant to be

[stone orbit]

everything else depends on it implicitly

[vale pelican]

a package that is a dependency of every other package without being listed

we used to use it for mlibc

but some time ago we just switched to listing mlibc explicitly everywhere

[stone orbit]

i like that but how many times have i been screwed over by typing it correctly i cannot count

[hexed solstice]

so that's why it happened then

mlibc-headers is also marked an implict package

thus keyronex-kernel-headers depended on it, while it depended on keyronex-kernel-headers

[stone orbit]

yeah that sounds about right

the only annoying this is that kernel implicitly depends on mlibc :p

but you build it anyway so it should be ok™️

i guess thats the reason for explicitly listing mlibc everywhere

though i feel like its super annoying if you have thousands of packages...

[vale pelican]

at that point a proper dsl would help more

[hexed solstice]

maybe that day will eventually come

[vale pelican]

like being able to generalize and abstract away build systems, so for a meson project you just list the options to set and tell it to go do it's thing

[stone orbit]

id love to build something like that but i dont have enough experience with build systems to actually do it

[hexed solstice]

pkgsrc is quite like that but building your own pkgsrc is very difficult

it's nearly 30 years of crystallised effort

[vale pelican]

i discussed a prototype for such a dsl (based on scheme) in the #misc-dev channel in managarm server but it wasn't as liked as i hoped it'd be :^)

[stone orbit]

could you link it here please?

i might just go ahead and implement it and play around with that since i have nothing better to do lol

[vale pelican]

#678613670980222987 message

it's only a prototype, and was designed in a way that it can just be implemented with scheme syntax-rules (with syntax-parameterize etc) or such

[stone orbit]

looks interesting but syntax is a bit confusing

[vale pelican]

yeah without prior scheme experience it's probably a bit weird

[hexed solstice]

it's up there with some of the better build system formats

[vale pelican]

in short, (a b c) vs '(a b c) is that the former is an application of a with args b, c while the latter is a quoted list, with 3 elements: symbols a, b, c

and ``(a b ,c)is a quasiquoted list, wherea bremain as symbols, but,cevaluatesc` and substitutes the result of the evaluation into it's placee

[hexed solstice]

i do like pkgsrc but it's quite 'magical' and part of that comes from it being Makefiles: https://github.com/NetBSD/pkgsrc/blob/trunk/lang/zig/Makefile

[stone orbit]

diff --git a/userland.yml b/userland.yml
index 4b52a1e..87ee1f8 100644
--- a/userland.yml
+++ b/userland.yml
@@ -521,7 +521,6 @@ packages:
         - '--with-system-expat'
         - '--disable-ipv6'
         - '--without-ensurepip'
-        - '--with-build-python=@THIS_BUILD_DIR@/python3.11'
         environ:
           CONFIG_SITE: '@THIS_BUILD_DIR@/confsite'
           PKG_CONFIG_SYSROOT_DIR: '@BUILD_ROOT@/system-root'
diff --git a/x11.yml b/x11.yml
index fd3fe42..1679c41 100644
--- a/x11.yml
+++ b/x11.yml
@@ -1217,6 +1217,8 @@ packages:
         - '--disable-static'
         - '--with-sysroot=@SYSROOT_DIR@'
         - '--with-xkb-output=/var/lib/xkb'
+        - '--with-xkb-bin-directory=/usr/bin'
+        - '--with-xkb-path=/usr/share/X11/xkb'
         - '--with-fontrootdir=/usr/share/fonts/X11'
         - '--enable-xorg'
         - '--enable-xv'
```that is my final patch after latest commit, could you also look into creating /dev with some make rule or xbstrap?

[vale pelican]

there's also ,@c which, if c is a list, would splice the list elements in, so ``(1 2 ,@(3 4))=>(1 2 3 4)(and not(1 2 (3 4))` for regular unquote)

[stone orbit]

also what is the dot in (summary . "blah blah")?

[vale pelican]

cons cell

summary is the car and the string is the cdr

[hexed solstice]

doing so, cheers, i will also make an anti-backtrace patch

[vale pelican]

normally a list is just a chain of cons cells where every cdr is a cons cell

[stone orbit]

good idea, i forgot about that one 😄

[vale pelican]

but for (a . b) the cdr is just b

as a result you can also form lists like (1 . (2 . (3 . ()))) (=> (1 2 3 4))

[hexed solstice]

i really need to figure out why xeyes doesn't work when XPRESENT is defined in its config.h also

very strange thing

[stone orbit]

it might be something to do with timers?

[vale pelican]

a cons cell is basically a linked list node, except for the improper case ((1 . 2)) next points to 2 itself and not a list node containing 2

[hexed solstice]

that's a distinct possibility, what led you to think that?

[stone orbit]

i imagine that present is driven by some kind of timer, so application that depend on that just wait for the server to give them that information

i have no idea how xorg really works so this is a wild guess

[hexed solstice]

oh and XTHREAD also breaks input iirc, there is some very strange bug in my epoll

[stone orbit]

but im thinking maybe the server isnt being properly waken up? or the timer never goes off for some reason

also i see that on the screenshot in readme you managed to move the xeyes window, how? i couldnt get mouse input to work :(

[vale pelican]

the reason for using this syntax here is that a list of improper cons cells ((a . 1) (b . 2)) is typically used as an association list, and as such there's an assoc procedure to look for a specific key in it

[hexed solstice]

this is the fault of the XTHREADS thing again i think

[stone orbit]

oh, that explains it 😁

[hexed solstice]

epoll is on my agenda for a robust rewrite in the near future

maybe even with support for EPOLLET and EPOLLONESHOT

it's annoying to lock the thing

the problem is that there exists a lock ordering struct epoll lock -> watched object's lock, which is fine when we are in an epoll syscall. it is pain and misery when we are in e.g. the TTY input routine and we want to wake up our epolls

i loathe lock ordering

locking in general actually. i think it is the hardest thing to do properly

[vale pelican]

one solution: no smp or kernel preemption :^)

[hexed solstice]

i have been seriously tempted at times

what's the threading story with managarm's posix server for that matter?

[vale pelican]

posix-subsystem is a single thread

it manages multiple clients via coroutines

and locking around some parts is probably missing, but most things are managed via shared_ptr, so at least the allocation/deallocation part should be safe if we decided to make it multithreaded

[stone orbit]

currently building and collecting a list of dependencies for debian based distros, ill make a pull request with some more build fixes and if you want i can also try setting up github actions

[hexed solstice]

cheers, this will be elpful

[stone orbit]

xbstrap: configure mlibc-headers [10/407]
xbstrap: Running ['meson', 'setup', '--cross-file', '@BUILD_ROOT@/crossfile.ini', '--prefix=/usr', '-Dheaders_only=true', '-Dbuildtype=debug', '-Dlinux_kernel_headers=@SOURCE_ROOT@/kernel/_sdkinc', '@THIS_SOURCE_DIR@'] (tools: ['crossfile'])

ERROR: Neither directory contains a build file meson.build.
xbstrap: Action configure of package mlibc-headers failed

[hexed solstice]

with the microkernel able to do quite a bit on its own it might even be OK performance-wise

[hexed solstice]

git submodules checked out?

[stone orbit]

oh god thats right

[hexed solstice]

i should maybe add a check to the makefile for that

[stone orbit]

yeah that sounds like a plan :p

[vale pelican]

yeah so far i don't think it's been a perf bottleneck

and besides there's still room to enable more compiler optimizations (we currently use --buildtype=debugoptimized)

[stone orbit]

ughhhh why does debian ship with ancient gcc??? ```
../../../mlibc/meson.build:56:1: ERROR: Value "c++20" (of type "string") for combo option "C++ language standard to use" is not one of the choices. Possible choices are (as string): "none", "c++98", "c++03", "c++11", "c++14", "c++17", "c++1z", "c++2a", "gnu++03", "gnu++11", "gnu++14", "gnu++17", "gnu++1z", "gnu++2a".

[vale pelican]

ancient meson

[stone orbit]

oh what?

[vale pelican]

not gcc

[stone orbit]

is that a meson thing?

even better

[vale pelican]

yeah meson maintains the list of standards it knows about itself

[stone orbit]

0.56.2

Stable release: 1.1.0 / 10 April 2023; 21 days ago

[vale pelican]

i'm currently on 1.0.1 :^)

[stone orbit]

💀

[vale pelican]

aka -g -Og

[hexed solstice]

i think i am building with -O0 at present

whatever is default for meson

[vale pelican]

yeah i think the default is the debug buildtype, which iirc is -g -O0

[hexed solstice]

wow

keyronex flies with -O2 compared to without

[stone orbit]

i would be surprised if there wasnt a noticeable change honestly

its incredible how big the difference is

and -O2 -g still produces great debug info!

especially with C, cant say the same for C++ but thats expected

[hexed solstice]

it makes a very meaningful difference to fork() performance

[stone orbit]

and it also serves as a great validation for all your inline assembly statements :^)

[hexed solstice]

i do it very non-optimally because i cba at the time implementing a proper protect-page-range function

one thing btw

i changed the python recipe and now i can't run python

it generated a linux binary

[stone orbit]

huh??

what the hell

[hexed solstice]

i thought i'd try python under -O2 keyronex and mlibc complained "No libm.so.6", and i thought, "oh ffs, i've blown it, there's some undefined behaviour somewhere and now it can't find files"

but apparently not

i don't have a libm.so.6 at all, python just built for linux

i fear python doesn't support --target and instead must have --host and --build explicitly specified

[stone orbit]

--target=xxx should work...

well that would suck

lets see what lfs does

hmm i see, lfs doesn't cross compile python

they compile it in a chroot with cross compiled tools

that is annoying

[hexed solstice]

i just checked and it builds fine with --build=x86_64-netbsd specified even when i build on fedora

[stone orbit]

that is interesting, i guess ill just take that

[hexed solstice]

and with a host-python recipe, the with-build-python ugliness is reduced

[stone orbit]

apparently you need --with-build-python

still

oh i guess i can use @BUILD_ROOT@ to dig into the build host python directory

ill see if it builds with the new recipe and continue building on ubuntu

should i bundle the python stuff in my pr?