Astral | Operating System Development | Page 13

analog berry Sep 10, 2024, 10:27 PM

#

it probably isnt a few seconds long tho

grand shadow Sep 10, 2024, 10:28 PM

#

yeah

#

in qemu it was like a small pop at the start iirc

#

will finish eating dinner and report if same happens on my desktop

analog berry Sep 10, 2024, 10:29 PM

#

would be nice to debug that poweroff as well

grand shadow Sep 10, 2024, 10:29 PM

#

I mean thats on oberrow's computer

#

so I need oberrow to exist

analog berry Sep 10, 2024, 10:29 PM

#

lol

#

i dont think he actually exists

grand shadow Sep 10, 2024, 11:11 PM

#

@carmine swallow I actually thing it might be playing some unitialized memory at the start

#

Same thing on my desktop, HOWEVER

#

It played a bit of the song before actually playing the start (I had to reboot the pc cuz I forgot to connect)

#

And it didnt finish the song too

#

grand shadow Sep 10, 2024, 11:12 PM

#

grand shadow It played a bit of the song before actually playing the start (I had to reboot t...

So I assume it was playing some unitialised memory

#

Cating it again just says starting stream and gets stuck there

analog berry Sep 10, 2024, 11:13 PM

#

grand shadow <@379250891838193667> I actually thing it might be playing some unitialized memo...

Any idea why?

grand shadow Sep 10, 2024, 11:14 PM

#

No idea I havent looked much into the hda driver, but this behaviour makes me think it is

#

I played a bit of it, realized I had the headphone plugged into the wrong port, rebooted, played it again, it played a wonky version of the middle of the song for a few seconds, then played the actual song start

#

so this is just my guess

#

holy shit I think my hda card is stuck

#

windows does not initialise it

#

or whatever

#

there is no option for it

#

loooooool

#

wtf did you do qwinci

#

lmaoooo

#

lemme see if cutting power helps it

#

Ok cutting power and turning it on again fixed it

#

actually fucking crazy tho

grand shadow Sep 10, 2024, 11:52 PM

#

qemu seems to be trying to write to read only memory here

grand shadow Sep 11, 2024, 12:14 AM

#

it seems like it tries to write to the first page of libx11 which uh what

grand shadow Sep 11, 2024, 1:27 AM

#

bruh

#

gnu hello doesnt build anymore

#

astral has fallen billions must use windows

grand shadow Sep 11, 2024, 1:44 AM

#

I dont get it? it detects libiconv, adds it to $LIBS in the configure script but it just doesnt pass to the makefile???

grand shadow Sep 11, 2024, 2:11 AM

#

so LIBICONV is passed to the makefile but it just doesn't add it to the link

#

what the fuck?

#

INTL is passed to the ld

#

maybe its some dependency memes

#

it does depend on libiconv

#

which makes me wonder why is libiconv not adding???

#

manually passing liconv works

#

ugh I think this is happening because of it direclty just linking the fucking .so

#

instead of being normal and passing -l

#

so the linker refuses to add the dependencies to intl because it thinks its an object so you have to pass it directly

#

nvm still happens

#

it only happens in hello too?

#

doing just -lintl outside of it works

#

just removing the libiconv dependency from intl

#

idc anymore

#

if it built it built

#

now it complains about undefined symbols to iconv in striconv.c in hello

#

aaaaaaaaaaaaaaaaaaaaaaaaaaaaa

#

stupid fucking autotools wont add the -liconv

fleet osprey Sep 11, 2024, 3:02 AM

#

this must be the least frustrating time I've seen anyone have using autotools meme

grand shadow Sep 11, 2024, 3:03 AM

#

@gritty shore looking into the server it seems you ran into this shit while building gnu hello, how did you fix it (if you even remember cuz its been a while)

#

gonna build some other package and see if its not a gnu hello autotools bug

carmine swallow Sep 11, 2024, 5:08 AM

#

grand shadow actually fucking crazy tho

yeah that's weird, I have seen the codec get stuck when you do passthrough but not after rebooting

#

as for the weird issue at the start I think I might know why, I only queue one page of data before starting the stream and the next interrupt is after that one page has been played so it writes to the same page that the controller is currently playing

analog berry Sep 11, 2024, 7:33 AM

#

grand shadow wtf did you do qwinci

Lmao

grand shadow Sep 11, 2024, 9:36 AM

#

carmine swallow as for the weird issue at the start I think I might know why, I only queue one p...

Hmmmmm

grand shadow Sep 11, 2024, 9:37 AM

#

grand shadow gonna build some other package and see if its not a gnu hello autotools bug

Forgot to mention but while configuring smth else I ran into an assert in ext2 so I will investigate it when Im home today

#

Due to my rmdir() being broken

#

I still do think its a bug on gnu hello's makefile stuff when running with libintl and libiconv

#

Since configure detects libiconv just fine and even passes it to the makefile but doesnt have it be in the actual rule

#

I will find out when I build another program with the autoconf GETTEXT stuff

grand shadow Sep 11, 2024, 4:38 PM

#

carmine swallow as for the weird issue at the start I think I might know why, I only queue one p...

Did you find out what the issue with your computer's card was?

carmine swallow Sep 11, 2024, 4:39 PM

#

no but I didn't really look into it that much yet (or to neither one of the ones that I own really)

grand shadow Sep 11, 2024, 4:40 PM

#

Ahh

#

I see

carmine swallow Sep 11, 2024, 4:40 PM

#

and the queueing stuff is somehow broken in qemu now too lol

#

as in that it plays a little bit of the song and then starts over playing it properly

grand shadow Sep 11, 2024, 4:41 PM

#

Hmm same thing that happened on my desktop

carmine swallow Sep 11, 2024, 4:42 PM

#

yeah I am not sure what is it caused by, it wasn't the buffer underflow that happened at the start because I fixed that

grand shadow Sep 11, 2024, 11:04 PM

#

implementing checking if directories are empty on tmpfs and ext2 on an unlink (I had forgor)

grand shadow Sep 11, 2024, 11:34 PM

#

back to trying to build some random gnu package to see if gettext works

#

´when do you think they'll turn it off lol

fleet osprey Sep 11, 2024, 11:35 PM

#

probably never

#

it takes basically nothing to run an FTP server

#

maybe like, 1MB of RAM

grand shadow Sep 11, 2024, 11:37 PM

#

it is year 2035, the ftp server of the gnu project has had the "we will probably shut this down in the future" notice for 14 years

#

when mkdirat mode

#

wait

#

wtf?

#

something isnt right

#

I think someone forgot to remove the print lol

fleet osprey Sep 11, 2024, 11:39 PM

#

yeah, seems like it

grand shadow Sep 11, 2024, 11:40 PM

#

@prime juniper literally unusuable pls fix

#

one day I will go through the output of configure and fix everything I can there

#

theres shit like dup2 works... no

#

lets see, grep configured

#

building it rn

#

it has the autoconf GETTEXT thing in configure.ac

#

same one that gnu hello has

#

and it built

#

thus, I will say that the gnu hello thing is their bug and not mine and I will not care

#

anyways the built grep works nice, which is neat

#

and e2fsck returns no issues

#

time to fix some other issues

fleet osprey Sep 11, 2024, 11:55 PM

#

🔥
tons of fixes today

grand shadow Sep 12, 2024, 12:49 AM

#

think my timers have some issue wrt removing entries

#

the following code breaks:
alarm 1 second
nanosleep for 1 nsec
alarm 1 second
nanosleep for eternity

#

the second alarm never goes off an my timer_remove returns some crazy value

#

so I am looking into that

fleet osprey Sep 12, 2024, 1:11 AM

#

well, at least if you can replicate it, you have a starting point
better than most of my bugs meme

wild marsh Sep 12, 2024, 2:45 AM

#

grand shadow wait

inb4 dennis meant to put the print in the managarm sysdep

prime juniper Sep 12, 2024, 5:35 AM

#

Whoops

grand shadow Sep 12, 2024, 10:51 AM

#

One day mkdirat will not ignore its mode and when that happens it will be the year of the managarm desktop

craggy tide Sep 12, 2024, 11:14 AM

#

lol

prime juniper Sep 12, 2024, 12:03 PM

#

grand shadow One day mkdirat will not ignore its mode and when that happens it will be the ye...

Aight bet. Sounds like I know my next PR meme

grand shadow Sep 12, 2024, 12:37 PM

#

@analog berry if its not too much trouble, can you give me a rundown of how acpi devices work, some examples of cool devices, how to talk to them, how to enumerate them on uacpi, etc? I want to redo how I do drivers soon and having acpi there from the start would be nice but I know like nothing about acpi

analog berry Sep 12, 2024, 12:38 PM

#

u can look at osdev wiki examples on how u can enumerate devices

#

TLDR devices have _HID/_CID that identify what they are, _CRS for getting the list of resources they provide or consume (e.g. port numbers, irqs, memory ranges, etc), other device specific methods, power states, etc

#

the wiki example is pretty good overview

grand shadow Sep 12, 2024, 12:40 PM

#

Ill look into it, thanks

analog berry Sep 12, 2024, 12:44 PM

#

cool, lemme know if u have any questions

grand shadow Sep 12, 2024, 12:50 PM

#

Will do

grand shadow Sep 12, 2024, 12:50 PM

#

grand shadow <@353593948523986945> if its not too much trouble, can you give me a rundown of ...

Itll also let me clean up the init code a bit, I might also do it for the filesystem code

grand shadow Sep 12, 2024, 1:58 PM

#

@analog berry does acpi need pci to initialise everything or can I use it to find the pci bus

carmine swallow Sep 12, 2024, 2:01 PM

#

grand shadow <@353593948523986945> does acpi need pci to initialise everything or can I use i...

namespace load potentially needs pci yeah

analog berry Sep 12, 2024, 2:02 PM

#

namespace load only requires access to pci bus 0

#

the rest can be iniitalized later

#

namespace initialize requires access to the entirety of pci

grand shadow Sep 12, 2024, 2:03 PM

#

I assume theres anoter table that isnt mcfg that says pci is real then?

analog berry Sep 12, 2024, 2:03 PM

#

the order:

uacpi_initialize
read MCFG
uacpi_namespace_load
find pci busses
uacpi_namespace_initialize

grand shadow Sep 12, 2024, 2:04 PM

#

Or how does it work in legacy pci

analog berry Sep 12, 2024, 2:04 PM

#

i mean if uacpi is accessing it then yes it does exist automatically

#

firmware wouldnt be trying to talk to it if it didnt exist

#

if no mcfg assume legacy pci if its ever accessed

grand shadow Sep 12, 2024, 2:05 PM

#

Alright ty

analog berry Sep 12, 2024, 2:05 PM

#

npnp

grand shadow Sep 12, 2024, 11:01 PM

#

grand shadow the following code breaks: alarm 1 second nanosleep for 1 nsec alarm 1 second na...

integer overflow in the timer code that converts microseconds to the timer value, oops!

#

the nanosleep for eternity would then proceed to insert an entry into the past

#

inventing time travel one bug at a time

grand shadow Sep 13, 2024, 1:07 AM

#

my ppoll had broken signal handling, oops

#

fixed that now and make -j2 is faster again

grand shadow Sep 13, 2024, 1:39 AM

#

I also need to revisit how I keep track of time

#

but that will come after I redo my driver stuff

#

qemu is slow as shit because as it turns out reading out of the hpet every time you want to know what time it is isnt that good lol

#

like actually crazy

#

so it might not even have been my slow as shit I/O but just fucking HPET

#

I love footguns that only appear like a year and a half later

robust geyser Sep 13, 2024, 1:42 AM

#

just keep time in a global variable that you update on every timer interrupt

#

and map the page containing that variable read-only in every userspace process :^)

#

i love virtual memory for being able to do stuff like that

grand shadow Sep 13, 2024, 1:47 AM

#

yeah I was probably thinking of something like that

#

my timer is one shot but I could figure something out

robust geyser Sep 13, 2024, 1:48 AM

#

good enough, no?

#

just read hpet into that variable

#

hm actually you end up with the same problem but in a different place lol

#

yeah you might need some proper time keeping here

grand shadow Sep 13, 2024, 1:50 AM

#

I should find out the average time it takes to access the hpet

#

apparently its like 1 us?

wild marsh Sep 13, 2024, 11:53 AM

#

check for invariant tsc and just use the tsc as the global time variable :^)

grand shadow Sep 13, 2024, 10:21 PM

#

so what I want from the new driver stuff I will make will be:
-have buses discover hardware and other buses under themselves (architecture specific root bus -> acpi bus -> pci bus -> virtio bus -> virtio block device, for example) and call to initialize them (by looking into a section marked for that specific bus to look for drivers in)
-have handling for buses to do hotplug for connection and removal on devices and buses under themselves via a callback or something
-have handling for deinitialization calls on system power off

#

anything else that would be interesting to have?

robust geyser Sep 13, 2024, 10:31 PM

#

kernel modules you can load and unload at runtime

#

:^)

analog berry Sep 13, 2024, 10:55 PM

#

grand shadow so what I want from the new driver stuff I will make will be: -have buses discov...

Nice

obsidian pivot Sep 13, 2024, 11:23 PM

#

robust geyser kernel modules you can load and unload at runtime

It's not even that hard

robust geyser Sep 13, 2024, 11:24 PM

#

its not that hard if you are writing your kernel from scratch and not trying to retrofit*

obsidian pivot Sep 13, 2024, 11:24 PM

#

Just a bit of elf funniness

grand shadow Sep 14, 2024, 12:20 AM

#

robust geyser kernel modules you can load and unload at runtime

sure

fleet osprey Sep 14, 2024, 12:40 AM

#

I'll add to this

#

automatically loadable kernel modules

#

so if a new device appears, it can know what driver to load for it, and do so automatically

obsidian pivot Sep 14, 2024, 11:01 AM

#

fleet osprey so if a new device appears, it can know what driver to load for it, and do so au...

So PnP

supple hatch Sep 14, 2024, 11:02 AM

#

obsidian pivot So PnP

pnp

obsidian pivot Sep 14, 2024, 11:02 AM

#

Plug n play

supple hatch Sep 14, 2024, 11:02 AM

#

plug n poop

tribal dust Sep 14, 2024, 12:01 PM

#

poop n play

obsidian pivot Sep 14, 2024, 2:57 PM

#

pooping play

supple hatch Sep 14, 2024, 6:43 PM

#

@grand shadow catting /dev causes a page fault

#

lol

grand shadow Sep 14, 2024, 7:02 PM

#

oops

#

I think I forgot to add a check lmao

supple hatch Sep 14, 2024, 7:05 PM

#

lol

supple hatch Sep 14, 2024, 7:06 PM

#

grand shadow I think I forgot to add a check lmao

i will cat /

#

memedown

#

cat the root directory

#

lol

#

CAT THE WORLD

obsidian pivot Sep 14, 2024, 7:36 PM

#

uhh

#

@grand shadow

#

yea chown does nothing

#

same with chmod

#

yeah the ext2 driver might somehow be broken?

#

I made an nvme disk

#

with an ext2 partition on the first partition

#

and it doesn't like it

#

"bad signature"

#

the debugfs tool is fine with the partition

#

unless I've done something wrong

grand shadow Sep 14, 2024, 7:56 PM

#

obsidian pivot uhh

Wtf

grand shadow Sep 14, 2024, 7:56 PM

#

obsidian pivot the `debugfs` tool is fine with the partition

Did you set the boot device to nvme0n1p1

obsidian pivot Sep 14, 2024, 7:56 PM

#

grand shadow Sep 14, 2024, 7:57 PM

#

By default it boots from the disk and not a partition

obsidian pivot Sep 14, 2024, 7:57 PM

#

grand shadow Sep 14, 2024, 7:57 PM

#

obsidian pivot

Yeah change the device in the limine entries

#

To nvme0n1p1

grand shadow Sep 14, 2024, 7:59 PM

#

obsidian pivot uhh

Oh yeah uh tiny detail I think i hvent fixed yet

#

I think the tmpfs root has mode 0 by default

#

Just a very tiny detail meme

#

@obsidian pivot did it work

obsidian pivot Sep 14, 2024, 8:14 PM

#

I'll be back in 1 and a half hour

grand shadow Sep 14, 2024, 8:14 PM

#

mkay

grand shadow Sep 14, 2024, 8:42 PM

#

I have fixed the tmpfs meme and the devfs memes on the latest 2 commits

fleet osprey Sep 14, 2024, 9:41 PM

#

nice

obsidian pivot Sep 14, 2024, 10:12 PM

#

@grand shadow

#

when booting from nvme0n1p1

grand shadow Sep 14, 2024, 10:16 PM

#

Hm

obsidian pivot Sep 14, 2024, 10:16 PM

#

nvm

grand shadow Sep 14, 2024, 10:16 PM

#

Can you boot from the initrd and mount the nvme disk

#

Oh

obsidian pivot Sep 14, 2024, 10:16 PM

#

I don't have init there

grand shadow Sep 14, 2024, 10:16 PM

#

Lmao

obsidian pivot Sep 14, 2024, 10:16 PM

#

on my partition

#

how I mount

#

mount?

grand shadow Sep 14, 2024, 10:17 PM

#

mount -d /dev/nvme0n1p1 /mnt ext2

#

Iirc

#

Just run mount and itll say what you need

obsidian pivot Sep 14, 2024, 10:18 PM

#

that was it

obsidian pivot Sep 14, 2024, 10:39 PM

#

#

@grand shadow check this out

grand shadow Sep 14, 2024, 10:40 PM

#

obsidian pivot

Holy shit I agree

obsidian pivot Sep 14, 2024, 10:41 PM

#

if anyone was wondering how you can make your own

#

https://makesweet.com/my/heart-locket#

MakeSweet

Heart Locket

Heart Locket: a heart-shaped locket, which opens to reveal whatever you choose to put in it - perhaps two photos o ...

obsidian pivot Sep 14, 2024, 10:44 PM

#

grand shadow Holy shit I agree

btw what bg colour do you use for astral's console

grand shadow Sep 14, 2024, 10:44 PM

#

obsidian pivot btw what bg colour do you use for astral's console

I forgor lemme get it

grand shadow Sep 14, 2024, 10:45 PM

#

obsidian pivot btw what bg colour do you use for astral's console

uint32_t defaultbg = 0x1b1c1b;
uint32_t defaultfg = 0xffffff;

obsidian pivot Sep 14, 2024, 10:45 PM

#

grand shadow Sep 14, 2024, 10:45 PM

#

Capybara

obsidian pivot Sep 14, 2024, 10:45 PM

#

grand shadow uint32_t defaultbg = 0x1b1c1b; uint32_t defaultfg = 0xffffff;

https://tenor.com/view/mine-now-steal-take-grab-mine-gif-13196166

Tenor

Mine Now

▶ Play video

supple hatch Sep 14, 2024, 10:47 PM

#

obsidian pivot https://makesweet.com/my/heart-locket#

#

https://makesweet.com/f/b/9/9/b99a2a89-13c2-42ba-ab4c-d4c3007338c4.gif

obsidian pivot Sep 14, 2024, 11:03 PM

#

obsidian pivot https://tenor.com/view/mine-now-steal-take-grab-mine-gif-13196166

took way to long to steal

#

I meant borrow

split gazelle Sep 14, 2024, 11:43 PM

#

supple hatch <@215823874615541761> catting /dev causes a page fault

fun fact: you can cat /dev in some bsds

#

even macos

#

for some reason

supple hatch Sep 14, 2024, 11:43 PM

#

does it crash it

split gazelle Sep 14, 2024, 11:43 PM

#

no

obsidian pivot Sep 14, 2024, 11:43 PM

#

what does it do

split gazelle Sep 14, 2024, 11:44 PM

#

it returns the raw ufs directory structure of /dev

#

back in the old unix days, everything is literally just a file

#

to the point you can cat directories

#

some bsds and sysv derivatives have removed it on all directories but /dev because some older programs needs it

grand shadow Sep 15, 2024, 2:26 PM

#

while I figure out the new driver stuff I should do something very silly (make smp not suck hard because jesus christ)

analog berry Sep 15, 2024, 2:40 PM

#

Truee

grand shadow Sep 15, 2024, 2:42 PM

#

and after that implement sysconf and after that have glxgears run multicore

#

since it checks sysconf for the number of processors online

prime juniper Sep 15, 2024, 5:09 PM

#

Something Managarm should also implement (tho we have some other scheduler memes to fix for that to be useful first). @lean kettle how’s that thread migration going?

lean kettle Sep 15, 2024, 5:09 PM

#

prime juniper Something Managarm should also implement (tho we have some other scheduler memes...

It's something I should look at again at some point

#

Unless the floods get so bad that it's literally impossible for me to return I'll be in Austria again by the 17.

prime juniper Sep 15, 2024, 5:15 PM

#

Alright. I’ll remind you again later this week then and/or check up on if Austria is Netherlands 2.0 now meme

#

Enjoy your time in London until then

lean kettle Sep 15, 2024, 5:16 PM

#

prime juniper Enjoy your time in London until then

Am currently in Birmingham/Coventry right now lol

#

Family wanted me to extend me being in the UK on quite short notice and the Airbnb there was cheap and the friend I'm visiting knows the place and was also OK with staying with me for a bit longer lol

prime juniper Sep 15, 2024, 5:22 PM

#

Aight. Sounds nice. Enjoy!

lean kettle Sep 15, 2024, 5:22 PM

#

Thanks! NekoRave

grand shadow Sep 15, 2024, 5:41 PM

#

prime juniper Something Managarm should also implement (tho we have some other scheduler memes...

wait managarm still doesnt have sysconf?

#

ohh you mean smp

prime juniper Sep 15, 2024, 5:48 PM

#

We have smp but thread migration might be meh. Sysconf is mostly stubbed (with the same defaults posix has)

grand shadow Sep 15, 2024, 6:40 PM

#

just added the include dependency stuff to my makefile 2024 is the year of the astral desktop

grand shadow Sep 15, 2024, 6:41 PM

#

prime juniper We have smp but thread migration might be meh. Sysconf is mostly stubbed (with t...

better than me who just doesnt bind threads to cpus unless they specifically ask for it meme

#

@carmine swallow @robust geyser pull and do a make clean-kernel all to generate the .d files

#

it finally happened, I stopped being lazy and made it better

robust geyser Sep 15, 2024, 7:21 PM

#

good stuff

#

ty

grand shadow Sep 15, 2024, 8:49 PM

#

and now I have changed all _cpu()->thread to current_thread()

#

I will also rename _cpu() to current_cpu() because something something reserved

analog berry Sep 15, 2024, 8:50 PM

#

_cpu is perfectly fine

#

__cpu or _Cpu is reserved I think

#

But your version sounds better definitely

grand shadow Sep 15, 2024, 8:51 PM

#

yeah current_cpu would also be more readable

gritty shore Sep 15, 2024, 8:52 PM

#

curcpu() superiority

grand shadow Sep 15, 2024, 8:53 PM

#

grand shadow and now I have changed all _cpu()->thread to current_thread()

think I still have some other things to change around the kernel though since all calls to _cpu() are racey if the thread can get preempted

winged basin Sep 15, 2024, 9:15 PM

#

linux loves underscores in its code but i resent them

#

everything is ___this() or ____that()

flat copper Sep 15, 2024, 9:17 PM

#

just have _

wild marsh Sep 15, 2024, 9:24 PM

#

foo calls _foo calls __foo calls ___foo

analog berry Sep 15, 2024, 9:24 PM

#

wild marsh `foo` calls `_foo` calls `__foo` calls `___foo`

They run out of ideas for naming internal static functions so they spam _______

grand shadow Sep 15, 2024, 9:25 PM

#

_____________variable_____________

#

fuck it

#

xX_____________variable_____________Xx

#

like those stupid nicknames people use in online games

flat copper Sep 15, 2024, 9:26 PM

#

wild marsh `foo` calls `_foo` calls `__foo` calls `___foo`

Foo -> FooEx -> FooEx2

grand shadow Sep 15, 2024, 10:04 PM

#

alright so that should be all the current_cpu() races fixed

#

now onto tlb shootdowns

#

one day I'm gonna learn how to use libX11 or whatever and write a small portable suika game clone

#

which reminds me, I have some silly X11 programs in my backlog I should port

prime juniper Sep 15, 2024, 10:34 PM

#

Might yoink them if they work on XWayland

grand shadow Sep 15, 2024, 11:56 PM

#

https://github.com/Mathewnd/Astral/commit/12b307554dd4e4c9258a28b76de9ec10f7b3bdb4

#

git push works

#

:')

#

I do like the tone of green flanterm picks

#

wait FUCK I did a typo

#

😭

obsidian pivot Sep 16, 2024, 12:01 AM

#

ps2moune

supple hatch Sep 16, 2024, 12:19 AM

#

im cold

grand shadow Sep 16, 2024, 1:40 AM

#

hi cold I'm mathewnd

tawny oak Sep 16, 2024, 1:49 AM

#

grand shadow <https://github.com/Mathewnd/Astral/commit/12b307554dd4e4c9258a28b76de9ec10f7b3b...

you pushed this commit from astral?

grand shadow Sep 16, 2024, 1:50 AM

#

tawny oak you pushed this commit from astral?

yeah

tawny oak Sep 16, 2024, 1:50 AM

#

haha very cool

grand shadow Sep 16, 2024, 1:50 AM

#

first commit pushed from inside it and I managed to make a typo which is very fitting

fleet osprey Sep 16, 2024, 3:48 AM

#

Based

#

Can't wait for the day where you can repeat this, then, still within Astral, open Discord and come brag in here meme

tribal dust Sep 16, 2024, 4:45 AM

#

do osdev in astral

#

build a DOS in real mode assembly then run it in bochs

grand shadow Sep 16, 2024, 9:34 AM

#

I have qemu and nasm and qemu seems to half work

grand shadow Sep 16, 2024, 9:34 AM

#

fleet osprey Can't wait for the day where you can repeat this, then, still within Astral, ope...

Thatd take webkitgtk which pain

#

Ill have to start crawling towards it slowly to get some progress done

#

Maybe after smp I can do a gtk port or smth

robust geyser Sep 16, 2024, 11:40 AM

#

gtk is piss easy

#

literally steal from vinix lol

#

it worked on lyre

#

it worked(works?) on vinix

#

almost zero effort port, as long as your kernel isnt poop

grand shadow Sep 16, 2024, 11:46 AM

#

I mean if it worked on lyre it probably works on vinix

neon crane Sep 16, 2024, 11:54 AM

#

grand shadow and now I have changed all _cpu()->thread to current_thread()

btw even if you didn't do this there was a way to fix all the current_cpu races

#

when fetching the current thread, you can simply mov rax, [gs:offsetofcurrentthreadincpustruct]

#

this is atomic so the gs base wont change during this fetch

#

you can also raise IPL/CR8 or disable interrupts

grand shadow Sep 16, 2024, 11:57 AM

#

neon crane when fetching the current thread, you can simply `mov rax, [gs:offsetofcurrentth...

This is what I am still doing in the back, gs points to the cpu structure and I do a gs relative mov

neon crane Sep 16, 2024, 11:58 AM

#

i see

grand shadow Sep 16, 2024, 11:59 AM

#

That was one of the most long standing smp bugs and Im glad I finally went and fixed it

#

Now I have to make my tlb shootdowns faster and then just fix whatever appears after

#

The tlb shootdowns are by far the biggest issue with smp

#

I do an ipi per page which is bad and I plan on just doing one per vmm_unmap etc

neon crane Sep 16, 2024, 12:00 PM

#

that's what i plan on doing, i already have the infra in place

#

honestly it's kind of weird

#

you can only do one TLB shootdown at once

#

there is rudimentary support for batching but

#

idk I'm pretty sure it's air tight at least

grand shadow Sep 16, 2024, 12:01 PM

#

How are you approaching them?

neon crane Sep 16, 2024, 12:03 PM

#

in a very stupid way, step 1 is to acquire a spin lock that guards the entire operation

#

step 2 is to invalidate the pages on the local CPU

#

step 3 is to acquire the CPU local TLB shootdown lock on all CPUs but our own, then fill in the details about the TLB shootdown in the CPU local struct

#

step 4 is to fire an IPI to every CPU but our own

#

this IPI will invalidate the specific pages and release the TLB shootdown lock

#

step 5 is to re-acquire and release the CPU local shootdown lock on all CPUs but our own after the IPI

#

step 6 is to release the main lock and return

#

its kinda stupid, I know

#

https://github.com/iProgramMC/Boron/blob/master/boron/source/ke/amd64/tlbs.c here's the code to what I just said

#

l84: i do notice that i also release the spinlock to the initiator PRCB's tlbs lock which should be fixed

#

i might add an assert in the future there & shit

grand shadow Sep 16, 2024, 12:06 PM

#

I do wonder if just reloading cr3 would be faster above a certain page count than looping through the whole thing invalidating pages

neon crane Sep 16, 2024, 12:07 PM

#

maybe

#

well you will have to do it if you invalidate any pages you have set the global bit on as reloading cr3 won't invalidate those

grand shadow Sep 16, 2024, 12:09 PM

#

Wrt to calling KeIssueTLBShootdown, do you just do that after the unmap operation (unmap -> shootdown -> release pages) or do you do it in an intermediate step (make pages inacessible -> shootdown -> unmap)?

neon crane Sep 16, 2024, 12:09 PM

#

As soon as things are unmapped but I plan on refactoring it

#

Technically you don't need to do it until you reuse that memory region

#

Practically you should

#

Such an optimization would probably only be valuable in kernel mode since the kernel trusts itself

#

Sorry I was a bit incoherent right now

#

What I'm trying to say is there's an optimization that you can do in kernel mode where you only issue a TLB shootdown when reusing a region of memory

#

And that it should be disabled in debug mode to catch potential use after free bugs which can manifest in a nasty way in release mode

grand shadow Sep 16, 2024, 12:11 PM

#

Yeah fadanoid had told me about it, I dont think I will do something like that because I dont trust myself lmaoo

neon crane Sep 16, 2024, 12:11 PM

#

that's why you disable it in debug mode 😄

grand shadow Sep 16, 2024, 12:12 PM

#

True

neon crane Sep 16, 2024, 12:12 PM

#

speaking of do you actually have a debug/release distinction?

#

where you disable certain sanity checks in release mode

grand shadow Sep 16, 2024, 12:12 PM

#

Nah I just have two defines for e9 logging and syscall logging

neon crane Sep 16, 2024, 12:12 PM

#

i see

grand shadow Sep 16, 2024, 12:13 PM

#

grand shadow Wrt to calling KeIssueTLBShootdown, do you just do that after the unmap operatio...

Realistically the unmap -> shootdown -> release would be better but Id need to find a way to keep track of all the pages and I dont want to add anything to struct page to not make it even bigger

neon crane Sep 16, 2024, 12:15 PM

#

grand shadow Realistically the unmap -> shootdown -> release would be better but Id need to f...

here's a way

#

clear the present bit in the PTE

#

and set another bit to tell the page fault handler that this page is to be freed and it's inaccessible

#

or something

grand shadow Sep 16, 2024, 12:18 PM

#

neon crane here's a way

I mean like for more userspacey stuff where you need to make sure there wont be an use after free

#

My original idea was to make them inacessible, shoot down and then unmap and release since the physical address will still be in the page tables

#

But that needs going through the ptes twice

neon crane Sep 16, 2024, 12:19 PM

#

so?

#

is that a problem

#

userspace typically rarely unmaps memory so I think it's fine

grand shadow Sep 16, 2024, 12:20 PM

#

Fair

wild marsh Sep 16, 2024, 1:08 PM

#

neon crane when fetching the current thread, you can simply `mov rax, [gs:offsetofcurrentth...

struct per_cpu __seg_gs *cur_cpu; :^)

winged basin Sep 16, 2024, 2:54 PM

#

wild marsh `struct per_cpu __seg_gs *cur_cpu;` :^)

i almost ended up doing this

#

but instead i've decided to do two sets of macros ({D,F}PCPU_{LOAD,STORE}). the F variant uses fixed offsets so they can be immdiates, while D uses variable offsets. i set up a .cpu_local section in the linker script and i arrange for a struct cpu_local to be positioned first (so that i can use offsetof on members of that in the F macros) followed by anything else defined with attribute section .cpu_local

#

the FPCPU_ are guaranteed to be atomic (amd64 %gs-relative, AArch64 %x18-relative, RISC-V %tp-relative, m68k they just directly touch the struct), the DPCPU are not guaranteed (they are on amd64 and aarch64, but this is impossible on riscv)

grand shadow Sep 16, 2024, 8:27 PM

#

ok so smp support had bitrotted a bit but I have fixed that
wrt to it working, it ran configure and make -j3 (running with 2 cores) without crashing which is a start
I am redoing my shootdowns atm which should speed it up quite a bit hopefully

grand shadow Sep 16, 2024, 10:41 PM

#

I wonder two things:
is a broadcast (all but self) ipi slower than sending an ipi to every single core individually? and do you need to EOI an ipi?

#

I cant find anything in the sdm about the later but I probably am just blind

#

I assume you do need an eoi though

obsidian pivot Sep 16, 2024, 10:44 PM

#

nope

#

at least I don't and it works fine

grand shadow Sep 16, 2024, 10:44 PM

#

oh bruh

obsidian pivot Sep 16, 2024, 10:44 PM

#

grand shadow I wonder two things: is a broadcast (all but self) ipi slower than sending an ip...

for the former I'd assume it depends on the cpu model

#

and design

grand shadow Sep 16, 2024, 10:45 PM

#

meh I might just send it indivually in my tlb shootdown, might allow me to do some other optimisations too

grand shadow Sep 17, 2024, 12:37 AM

#

well good news

#

it works

#

bad news

#

its still slow

#

qemu-prof.sh does say that it isnt in the tlb stuff

#

so I suppose thats just an astral skill issue somewhere

#

and glxgears segfaults meme

#

bad action

#

I assume its because of the changes I did to a vmm function

grand shadow Sep 17, 2024, 12:42 AM

#

grand shadow its still slow

wonder if this could be because I have one queue for all cpus

#

I wouldnt imagine there'd be that much contention at 2 cpus though?

grand shadow Sep 17, 2024, 1:09 AM

#

the oberrow curse finally got to me @obsidian pivot

obsidian pivot Sep 17, 2024, 1:10 AM

#

t'was bound to happen

#

gl on your journey

#

#1061407633745125397 message

#

found first message in astral thread

grand shadow Sep 17, 2024, 1:12 AM

#

oop wait

#

I think I found the issue

#

yeah

#

that was stupid

obsidian pivot Sep 17, 2024, 1:13 AM

#

what was it

grand shadow Sep 17, 2024, 1:13 AM

#

I <3 logic errors

obsidian pivot Sep 17, 2024, 1:13 AM

#

I love off-by-one bugs that cause two-weeks of debugging, two rewrites of an allocator, and your kernel earning the title "cursed" for the next year or two (it is cursed)

grand shadow Sep 17, 2024, 1:13 AM

#

obsidian pivot what was it

-                               newrange->start = (void *)((uintptr_t)range->start + range->size + difference);
+                               newrange->start = (void *)((uintptr_t)range->start + range->size);

#

glxgears still segfaultsaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

#

oh oops

#

I shouldnt program while tired

#

ok glxgears works single core at the normal fps

#

brO

#

2 core makes it

#

uh

#

why

#

WHY

#

mouse accelerated graphics I guess

#

the fuck

#

everything works at normal speed when I move the mouse

#

hm

#

its either my scheduler being sucky for smp (though still 2 cores makes it bad like that???) or some other skill issue

#

well

#

at least its stableish meme

#

considering most of the cpu time seems to be on the idle thread, I'm assuming its some other locking issue

grand shadow Sep 17, 2024, 2:08 AM

#

could it be a race in my poll system?

#

since most of the ipc goes through it

#

and this shit is ipc heavy

#

shit

#

it might be a lost wakeup?

#

                if (revents == 0 || spinlock_try(&desc->lock) == false || spinlock_try(&desc->wakeuplock) == false) {
                        removefromlist(&pending, iterator);
                        insertinheader(header, iterator);
                }

                if (desc->event == NULL) {
                        iterator->revents = revents;
                        desc->event = iterator;
                }

        spinlock_acquire(&desc->eventlock);
        if (desc->event) {
                spinlock_release(&desc->eventlock);
                interrupt_set(intstate);
                return 0;
        }

#

seems like so

#

it does a check for desc->event in poll_dowait() but the spinlock_try(desc->lock) check prevents it from reaching desc-event = iterator;

#

that'd probably also explain why moving the mouse around makes it happy again

#

now I just have to go back in time and see what reason I had for desc->event = iterator to be down there

#

wait no

#

the desc->event is still reached

#

I'm just tired

#

still seems to be missing some wake up......

#

printing a ! on the sched_wakeup in poll_event and a . before the yield in poll_dowait shows a few seconds of delay every once and a while between the . and the !

#

its like a second

#

whatever I'll figure it out tomorrow thats enough

carmine swallow Sep 17, 2024, 7:56 AM

#

btw I figured out one of the hda issues, I had set it so it only generates an irq after it has initially played two pages worth of data and the stream was started when there was two pages of data in the buffer and then it wouldn't generate an irq until it was done playing those two pages (and then it would be too late because the controller would already be playing the third page)

grand shadow Sep 17, 2024, 9:10 AM

#

Huge

#

Does it work on your machine now

#

It might do on mine

robust geyser Sep 17, 2024, 9:17 AM

#

cool stuff

#

i should try and build gcc with gcc-mlibc on linux to see if its really an mlibc allocator bug

grand shadow Sep 17, 2024, 9:17 AM

#

@analog berry moving here from managarm

Smp seems to be stable with the new tlb shootdowns (didnt crash for me lol even with configure and make -j) and the only issue I think is some lost wakeup in some event code or smth

grand shadow Sep 17, 2024, 9:17 AM

#

robust geyser i should try and build gcc with gcc-mlibc on linux to see if its really an mlibc...

Dennis did that and it works so

#

But

#

He did it on a newer version of gcc

analog berry Sep 17, 2024, 9:18 AM

#

grand shadow <@353593948523986945> moving here from managarm Smp seems to be stable with the...

Nice

robust geyser Sep 17, 2024, 9:18 AM

#

i can try on the same version as we tried with astral

grand shadow Sep 17, 2024, 9:18 AM

#

Would be cool

robust geyser Sep 17, 2024, 9:19 AM

#

yea no problem i just need to grab linux-mlibc

#

https://github.com/no92/linux-mlibc god bless

#

damn this shit old

#

gcc 11.2.0

#

oh astral is on 12.1.0

#

i thought gcc was on like 14.2 nowadays

#

ok this is pain, for some reason mlibc wants libgcc and libgcc wants mlibc?

grand shadow Sep 17, 2024, 9:43 AM

#

Libgcc moment

craggy tide Sep 17, 2024, 10:32 AM

#

robust geyser ok this is pain, for some reason mlibc wants libgcc and libgcc wants mlibc?

yeah?

#

the way Vinix/Gloire solve it is by just using libgcc-binaries

#

you can compile gcc twice otherwise, but it doesn't really improve much

prime juniper Sep 17, 2024, 10:53 AM

#

Yeah after working around that

#

I got a compile error in some gcov bs on Linux mlibc

carmine swallow Sep 17, 2024, 10:54 AM

#

grand shadow Does it work on your machine now

I didn't try yet

carmine swallow Sep 17, 2024, 10:55 AM

#

robust geyser ok this is pain, for some reason mlibc wants libgcc and libgcc wants mlibc?

I don't think mlibc needs libgcc for anything on x86_64 at least

#

so you could just make it not link against it pretty sure

grand shadow Sep 17, 2024, 11:08 AM

#

carmine swallow I didn't try yet

Ah damn

#

If you leave an iso here I can try it later on my machines

carmine swallow Sep 17, 2024, 11:41 AM

#

📎 astral.iso

#

I tried it on the laptop but still got no sound (which I kinda expected tbh)

grand shadow Sep 17, 2024, 11:53 AM

#

It could be some other issue on yours

#

Though I think it should be fixed on mine

#

When I get home Ill try it

grand shadow Sep 17, 2024, 12:12 PM

#

I should see if qemu is fixed now after fixing that vmm logic issue

#

Since iirc it was a bad action access that crashed it too

grand shadow Sep 17, 2024, 12:14 PM

#

grand shadow qemu seems to be trying to write to read only memory here

Yeah

craggy tide Sep 17, 2024, 5:24 PM

#

carmine swallow I don't think mlibc needs libgcc for anything on x86_64 at least

it does

#

ever since i added complex functions it does

carmine swallow Sep 17, 2024, 5:32 PM

#

craggy tide ever since i added complex functions it does

ah

craggy tide Sep 17, 2024, 5:49 PM

#

and in any case on other non-x86 archs you want it anyways, so...

grand shadow Sep 17, 2024, 7:30 PM

#

@carmine swallow works on my desktop fine

#

gonna test on laptop now

#

though ^Cing it and trying to cat it again hangs it

#

Laptop works too

#

Good work qwinci

grand shadow Sep 17, 2024, 7:37 PM

#

grand shadow though ^Cing it and trying to cat it again hangs it

this makes my audio card get stuck btw

#

same shit as last time lmao

carmine swallow Sep 17, 2024, 7:47 PM

#

grand shadow Laptop works too

nice

#

I have no idea about the card getting stuck tho KEKW

#

nor it hanging if you try to cat it again into there, that doesn't happen in qemu

grand shadow Sep 17, 2024, 7:54 PM

#

you can ctrl c and then cat again in qemu?

#

hmmm

#

funky

carmine swallow Sep 17, 2024, 7:55 PM

#

yeah you can

#

at least as far as I tried

grand shadow Sep 17, 2024, 7:55 PM

#

also did this fix the sdl delay

carmine swallow Sep 17, 2024, 7:56 PM

#

Ill have to try, there is some kind of issue with doing a full build with make now tho

#

grand shadow Sep 17, 2024, 7:58 PM

#

wait

#

did I push install-astral

carmine swallow Sep 17, 2024, 7:58 PM

#

ah there is not even a source for that

grand shadow Sep 17, 2024, 7:58 PM

#

oml I'm so dumb lmao

#

that was just a small test I had done

#

there pushed the removal

carmine swallow Sep 17, 2024, 8:18 PM

#

did I ever happen to say why did tyrquake get stuck in "initializing palettes"? I can just find me saying about it but nothing about what the problem was

grand shadow Sep 17, 2024, 8:28 PM

#

I dont thunk so

grand shadow Sep 17, 2024, 10:43 PM

#

disabling the setitimer system calls makes it go away

#

fun

supple hatch Sep 17, 2024, 10:43 PM

#

hi mathew

#

i exist !!!

grand shadow Sep 17, 2024, 10:46 PM

#

hello nyauxmaster 5.0

grand shadow Sep 17, 2024, 11:55 PM

#

hm yes poll is returning with revents = 0 for the fd

#

makes complete sense

#

yes

#

totally sane to return from the non-interrupted path where an event had to happen and you have a desc.event pointer to the data but it has revents 0

#

so there is a race here

#

it wasnt a race but me being an idiot

#

(the revents = 0 part)

#

seems like the xorg server is getting stuck in a setitimer call

#

for like a second

#

every once in a while

#

while fvwm is starting

#

that is progress on figuring out wtf is going on

#

my guess would be its stuck trying to schedule into the cpu where the itimer timer entry is on

#

honestly what I could do is temporarily raise the priority of the thread wanting to reschedule

#

and send an ipi to the other cpu to end the timeslice asap or something

#

that actually makes sense why its a bit laggy

#

x loves setitimer

#

I will add a sched_reschedule_on_cpu() function then that will handle optimising that shit

#

one thing I know for certain

#

smp on astral sucks lol

#

that's something I'll have to slowly work on

#

a proper smp scheduler, better locking, maybe even lockless things where I can fit it

grand shadow Sep 18, 2024, 12:34 AM

#

grand shadow my guess would be its stuck trying to schedule into the cpu where the itimer tim...

and small things like these that dont matter on uniprocessors but can make shit bad in multiprocessors

grand shadow Sep 18, 2024, 1:54 AM

#

and yeah seems like implementing the sched_reschedule_on_cpu() fixed that bullshit

obsidian pivot Sep 18, 2024, 1:55 AM

#

seems as if we're both dealing with our schedulers

#

gl and god speed

grand shadow Sep 18, 2024, 1:57 AM

#

the framerate is still unstable but at least its not whatever the fuck was happening before

#

2 cores

#

there is probably something else fucking me over

#

seems to slowly stabilize towards the upper 100s though

#

and its not dying so

#

smp seems stable

#

wonder if this random stuttering could really just be because of my scheduler not being written with per cpu queues

grand shadow Sep 18, 2024, 2:00 AM

#

obsidian pivot seems as if we're both dealing with our schedulers

schedulers are fun until they arent

#

progress is at least being made and that makes me happy

#

just has rough edges

fleet osprey Sep 18, 2024, 2:10 AM

#

grand shadow seems to slowly stabilize towards the upper 100s though

yeah, this is certainly an improvement over mouse-accelerated glxgears, lol

grand shadow Sep 18, 2024, 9:22 AM

#

Soon I will also change read, write and a few others to write directly to the page cache/device/pipe instead of the shitty way Im doing it rn

#

Think this just takes changing a few memcpys to the possible copy from user macro and then passing it directly

#

Ill need to see what to do wrt to readv/writev though

#

I forgot if it has any atomicity guarantees about the rw

#

It is

#

So Ill have to look into how I will do that

#

The socket read/write already takes in a descriptor about the operation that I can just slightly change if need be

#

Id also need to change the vfs_read/write page cache copy to support the new iov stuff

robust geyser Sep 18, 2024, 10:06 AM

#

just lock for the duration of the copy, wouldnt that work?

#

in case of readv/writev

#

or just hope that it is going to be fine™️

#

iirc llinux does that with a lot of things, like i think directory listings are supposed to be atomic?

#

but i think that linux doesnt care

#

its one of those stupid things which probably dont matter in real world tho

#

this might be a bigger problem, but who would writev and write to the same fd at the same time anyway?

#

or writev from two different threads

grand shadow Sep 18, 2024, 10:09 AM

#

My mind comes to named pipes, which you could write to from multiple processes and itd come out the other way in a way it should not

robust geyser Sep 18, 2024, 10:10 AM

#

ah yeha that makes sense actually

grand shadow Sep 18, 2024, 10:11 AM

#

I think just changing the vnode read/write functions to take in an iov and have a helper would be the better option

robust geyser Sep 18, 2024, 10:11 AM

#

i cant help much im not knowledgable enough beyond "just slap a mutex around it™️"

grand shadow Sep 18, 2024, 10:14 AM

#

Also pipe I/O has a guaranteed atomicity based on the size that I dont know how it interacts with readv/writev

#

Id assume it gets treated as just one big chunk

supple hatch Sep 18, 2024, 5:33 PM

#

grand shadow hello nyauxmaster 5.0

hi

grand shadow Sep 18, 2024, 8:59 PM

#

smp isn't very quick but it really seems stable now

#

the whole wget gnu hello etc etc make -j5 with 4 cores works fine

analog berry Sep 18, 2024, 9:02 PM

#

Should try it on the managarm 128 core pc

robust geyser Sep 18, 2024, 9:02 PM

#

you can build gnu hello again?

grand shadow Sep 18, 2024, 9:03 PM

#

robust geyser you can build gnu hello again?

well the intl meme still happens but meh

#

thats an issue with gnu hello's Makefile.am I am pretty sure

robust geyser Sep 18, 2024, 9:03 PM

#

also i should try smp too but i cba to build astral again meme

analog berry Sep 18, 2024, 9:04 PM

#

CBA?

grand shadow Sep 18, 2024, 9:04 PM

#

analog berry Should try it on the managarm 128 core pc

it'd probably boot but slow down to a halt

analog berry Sep 18, 2024, 9:04 PM

#

grand shadow it'd probably boot but slow down to a halt

Why is that

grand shadow Sep 18, 2024, 9:04 PM

#

idk smp is still not 100% ok in astral

#

like its slower for some reason which I am still trying to find

#

either a scheduler meme or an ipc meme or something

#

just remembered another optimisation I still dont do, wake up a core to run a thread when its woken up if theres a core idle

robust geyser Sep 18, 2024, 9:07 PM

#

you should try to schedule it on a core the thread has affinity for

grand shadow Sep 18, 2024, 9:08 PM

#

I dont have per core queues yet, which is why I want to just redo the whole thing

#

also this

#

man my bad design decisions really are biting me in the ass now

#

but its not unsalvageable like old astral (at least not yet)

winged basin Sep 18, 2024, 9:12 PM

#

grand shadow I dont have per core queues yet, which is why I want to just redo the whole thin...

whole scheduler you mean?

obsidian pivot Sep 18, 2024, 9:13 PM

#

probably

analog berry Sep 18, 2024, 9:13 PM

#

Inb4 astral rewrite

grand shadow Sep 18, 2024, 9:13 PM

#

yeah scheduler

#

this is how glxgears runs with 2 cores

#

it really is unacceptable

#

though its something I've been neglecting since forever so its better I start fixing it now than later

analog berry Sep 18, 2024, 9:14 PM

#

Since u only have one queue u have insane contention

grand shadow Sep 18, 2024, 9:14 PM

#

even with like two cores?

analog berry Sep 18, 2024, 9:15 PM

#

The lag is definitely contention I think

#

Where its coming from idk

#

U have lapic running on both right

grand shadow Sep 18, 2024, 9:15 PM

#

I think I might just be doing a lot of shootdowns too

grand shadow Sep 18, 2024, 9:15 PM

#

analog berry U have lapic running on both right

yeah

analog berry Sep 18, 2024, 9:16 PM

#

What are u shooting down lol

#

Are they unmapping stuff

grand shadow Sep 18, 2024, 9:16 PM

#

grand shadow I think I might just be doing a lot of shootdowns too

due to me not queueing up ranges like in that optimisation

robust geyser Sep 18, 2024, 9:16 PM

#

you only need to shootdown if mapping flags become more permissive

grand shadow Sep 18, 2024, 9:16 PM

#

analog berry Are they unmapping stuff

my read/write/recvmsg/whateverf being shit and doing a vmm_map and vmm_unmap

robust geyser Sep 18, 2024, 9:16 PM

#

-x or -w

analog berry Sep 18, 2024, 9:16 PM

#

grand shadow my read/write/recvmsg/whateverf being shit and doing a vmm_map and vmm_unmap

Bruh

robust geyser Sep 18, 2024, 9:16 PM

#

or if you unmap

#

so yeah...

#

i see

grand shadow Sep 18, 2024, 9:17 PM

#

which is also why I am gonna stop being a lazy fuck and actually make they go directly to/from where they need to go

analog berry Sep 18, 2024, 9:17 PM

#

Youre only sending tlb shootdowns if a process has multiple threads right

grand shadow Sep 18, 2024, 9:17 PM

#

analog berry Youre only sending tlb shootdowns if a process has multiple threads right

yeah

#

        bool do_shootdown = // do shootdown if
                current_thread() // scheduler is up
                && arch_smp_cpusawake >= 2 // and there are multiple cpus in the system
                && (page >= KERNELSPACE_START // and its either in the kernel
                || ((page == NULL || (page >= USERSPACE_START && page < USERSPACE_END)) // or in userspace...
                        && thread->proc && thread->proc->runningthreadcount > 1)); // in a process which has multiple threads running

analog berry Sep 18, 2024, 9:17 PM

#

Didn't know gears were multithreaded

grand shadow Sep 18, 2024, 9:17 PM

#

its because its shooting them down in the kernel

#

due to me not copying directly to the page cache or whatever

#

and the heavy amount of ipc

analog berry Sep 18, 2024, 9:18 PM

#

I dont understand

grand shadow Sep 18, 2024, 9:18 PM

#

like

#

every read/write/send/recv/whatever maps and unmaps some memory in the kernel temporarily to store the whole data

#

and if you do a lot of these you do a lot of shootdowns

#

because I dont queue them yet

analog berry Sep 18, 2024, 9:19 PM

#

Wait ur doing that literally in the kernel

#

Bruuuh

grand shadow Sep 18, 2024, 9:19 PM

#

yeah ik it sucks

analog berry Sep 18, 2024, 9:19 PM

#

Optimize this

grand shadow Sep 18, 2024, 9:19 PM

#

I am working on it rn lol

analog berry Sep 18, 2024, 9:19 PM

#

Nice

grand shadow Sep 19, 2024, 1:14 AM

#

ok, I've implemented enough for read(2) and write(2) to write directly to stuff and for stuff to go directly to the page cache too

#

noticeable improvement in glxgears performance for single core and startwm time for multicore

#

still missing recvmsg and sendmsg and other iov things though

#

that will be for tomorrow as I am too tired

#

used to be at like 200 for single core

#

now its almost 250

grand shadow Sep 19, 2024, 1:35 AM

#

every day that passes astral sucks a little less :D

obsidian pivot Sep 19, 2024, 1:35 AM

#

WWWW

grand shadow Sep 19, 2024, 1:56 AM

#

@carmine swallow btw I changed the rules for read() and write(), now they can be passed an userland buffer
to safely do them you can use the usercopy functions (in the driver case for read it'd be with the USERCOPY_POSSIBLY_FROM_USER() macro for example)
ringbuffer read/write/peek can also now copy to/from user and returns RINGBUFFER_USER_COPY_FAILED if it failed (you should return EFAULT if it does or handle it appropriately)
do note that this means you can take a page fault if youre copying in the ring buffer to/from user or using the user copy
I mitigated this in other places by exploiting the fact that my ringbuffer implementation apparently doesnt need a lock to synchronize reads and writes, it only needs a lock really to protect itself from multiple readers or multiple writers, which can be delegated to a blocking mutex if need be

just telling you tho cuz you're working on the hda driver and this was kinda a major change
tho I'd wait until I have the iov stuff fleshed out since I still need to deal with that and it'd change stuff a tiny bit meme

#

I hope to have smp not sucking so hard by monday hopefully

#

Uni starts again monday

#

And friday is a day off work

analog berry Sep 19, 2024, 2:08 AM

#

Good work

grand shadow Sep 19, 2024, 12:02 PM

#

The talk in nyaux reminded me that I need to revisit my timers and make them suck less

#

I suppose finishing the rest of this work for smp, and then work on timers and possibly a scheduler rewrite(?)

#

Idk if I need to rewrite the whole thing or change some of it

#

Defo clean up though

supple hatch Sep 19, 2024, 12:08 PM

#

i believe in u dwdw

obsidian pivot Sep 19, 2024, 12:11 PM

#

Kernel rewrite meme

supple hatch Sep 19, 2024, 12:11 PM

#

trl

#

do it do it do it

grand shadow Sep 19, 2024, 12:12 PM

#

Nah

#

Its not a lost cause yet

grand shadow Sep 19, 2024, 12:46 PM

#

Reading the ULE scheduler paper

grand shadow Sep 19, 2024, 1:33 PM

#

So far ULE seems simple enough and would possibly help increase responsiveness a ton on X.org

#

Though I still have a few pages to go its interesting

grand shadow Sep 19, 2024, 2:37 PM

#

@analog berry does acpi allow you to find the cpu topology of the system

winged basin Sep 19, 2024, 2:39 PM

#

grand shadow <@353593948523986945> does acpi allow you to find the cpu topology of the system

see the SLIT and SRAT tables

analog berry Sep 19, 2024, 2:56 PM

#

grand shadow <@353593948523986945> does acpi allow you to find the cpu topology of the system

Yes

grand shadow Sep 19, 2024, 3:15 PM

#

Bet

#

ULE seems simple enough and good

analog berry Sep 19, 2024, 3:19 PM

#

Ule?

grand shadow Sep 19, 2024, 3:21 PM

#

The freebsd scheduler

analog berry Sep 19, 2024, 3:34 PM

#

Ohh

winged basin Sep 19, 2024, 3:39 PM

#

it's very good

grand shadow Sep 19, 2024, 7:16 PM

#

IDK why but reading about ULE really hyped me up to rewrite my scheduler for some reason

grand shadow Sep 19, 2024, 8:19 PM

#

found a bug in recvmsg

#

turns out I was not returning the flags nor the control length field properly

#

and returning them properly also made me find a non initialized stack variable bug in localsockets

#

so thats one less worry now

fleet osprey Sep 19, 2024, 8:44 PM

#

nice

grand shadow Sep 19, 2024, 10:30 PM

#

working on a proper iovec abstraction now

grand shadow Sep 19, 2024, 11:17 PM

#

the goal for this would be to seamlessly replace the (buffer, size) in arguments for iovec_iterator_t and any ringbuffer_blahblah or usercopy_blahlblah calls with the new iovec calls

grand shadow Sep 20, 2024, 4:04 PM

#

I think the iovec_iterator interface is done

#

I will first implement it for sockets as a first test

#

Then once Im certain its working I will replicate it for the vfs

#

And devfs

grand shadow Sep 20, 2024, 9:56 PM

#

I should test to see if the bootsector snake clone I wrote works in astral qemu

#

at least that

#

also this is using the new iovec_iterator api for sockets

#

still havent checked the smp performance

#

better but still not satisfactory

grand shadow Sep 20, 2024, 11:59 PM

#

If I had to guess its the cache being wiped every time it moves cpus

#

Need to run a profiler after I switch everything to iovecs

grand shadow Sep 20, 2024, 11:59 PM

#

grand shadow If I had to guess its the cache being wiped every time it moves cpus

Cuz I have no per cpu queue

analog berry Sep 21, 2024, 12:00 AM

#

No need to profile, it is that lmao

gritty shore Sep 21, 2024, 12:39 AM

#

Pin tasks to a cpu

grand shadow Sep 21, 2024, 12:47 AM

#

I will rewrite the scheduler soon to do that

neon crane Sep 21, 2024, 8:08 AM

#

grand shadow at least that

@grand shadowcan you send me a complete iso?

#

i wanna see if i can compile & run boron on here

#

(nanoshell requires clang or 32-bit gcc and i dont know if you have either)

#

(but boron rn uses the system compiler)

grand shadow Sep 21, 2024, 12:45 PM

#

Gimme a sec

grand shadow Sep 21, 2024, 2:35 PM

#

@neon crane this one should be updated

obsidian pivot Sep 21, 2024, 2:37 PM

#

the message is a month old

#

but I'll just trust the iso is updated

grand shadow Sep 21, 2024, 2:37 PM

#

I update the image meme

grand shadow Sep 21, 2024, 3:57 PM

#

this work I'm doing of using scatter gather for all i/o will pave the way for clustered page in if I ever want to implement it

#

being able to cleanly iterate through the iovecs like its a stream and do stuff like memset/memcpy/ringbuffer rw/getting a page without much changes to existing code is nice

grand shadow Sep 21, 2024, 4:46 PM

#

alright, I think all devices now support the iovec_iterator

#

all I have to do is just hook and adapt the filesystem drivers now

robust geyser Sep 21, 2024, 4:48 PM

#

great job

#

i admire how efficiently you implemented it

neon crane Sep 21, 2024, 5:37 PM

#

grand shadow <@700663172226482254> this one should be updated

ty

#

what the fuck

#

i think the compositor died

grand shadow Sep 21, 2024, 5:55 PM

#

wtf

#

what did you do

neon crane Sep 21, 2024, 5:56 PM

#

idk lol

#

i think i was futzing around with the "restart" option

#

i really dont remember what led up to that point

neon crane Sep 21, 2024, 5:56 PM

#

neon crane i think i was futzing around with the "restart" option

this seems to restart only fvwm and not the system nor X.org

fleet osprey Sep 21, 2024, 5:57 PM

#

wtf
that's one interesting-looking way to fail

grand shadow Sep 21, 2024, 5:57 PM

#

fvwm3 probably died then

#

since the startwm script execs into fvwm

neon crane Sep 21, 2024, 5:59 PM

#

Also the clone process never seems to start

#

#

I wonder if it's because of qemu on windows' shitty network implementation

#

Other than using -display sdl (the default one is ass and wont keep the mouse locked)

#

I used the exact command line you sent

grand shadow Sep 21, 2024, 5:59 PM

#

did you run netd

#

netd vionet0

#

I dont start it automatically

neon crane Sep 21, 2024, 6:01 PM

#

oh

#

funny how it says that because the directory was empty

grand shadow Sep 21, 2024, 6:02 PM

#

wonder what it even tries to do

#

since rmdir works

neon crane Sep 21, 2024, 6:03 PM

#

#

this is uh

#

weird

robust geyser Sep 21, 2024, 6:03 PM

#

maybe it would be a good idea to use -nostdinc in boron

neon crane Sep 21, 2024, 6:04 PM

#

i should consider it

robust geyser Sep 21, 2024, 6:04 PM

#

so it doesnt try to pull in random system headers

#

lol

#

yeah you definitely should

neon crane Sep 21, 2024, 6:04 PM

#

https://github.com/osdev0/freestnd-c-hdrs-0bsd

#

but yknow

#

what IS the replacement for sys/cdefs.h here

#

its included by freebsd's sys/tree.h too

robust geyser Sep 21, 2024, 6:05 PM

#

https://github.com/openbsd/src/blob/master/sys/sys/cdefs.h

#

tldr a bunch of shit

neon crane Sep 21, 2024, 6:07 PM

#

ok no it works without that header lmao

#

oh i think it will get stuck at the python generate_symbols.py step

grand shadow Sep 21, 2024, 6:07 PM

#

y

neon crane Sep 21, 2024, 6:08 PM

#

nvm

#

seems like yo udo have python3

grand shadow Sep 21, 2024, 6:08 PM

#

yeah

neon crane Sep 21, 2024, 6:08 PM

#

its going!

#

that warning i gotta fix because KeInitializeThread always returns SUCCESS

fleet osprey Sep 21, 2024, 6:08 PM

#

[CC]\t
love that formatting

neon crane Sep 21, 2024, 6:08 PM

#

oh

neon crane Sep 21, 2024, 6:08 PM

#

fleet osprey `[CC]\t` love that formatting

Its supposed to, you know, actually print the tab character

grand shadow Sep 21, 2024, 6:09 PM

#

neon crane oh

did it not chmod +x

neon crane Sep 21, 2024, 6:09 PM

#

nope

#

probably my own fault as i develop on windows

grand shadow Sep 21, 2024, 6:09 PM

#

yeha

neon crane Sep 21, 2024, 6:09 PM

#

oh god i gotta call git again

grand shadow Sep 21, 2024, 6:10 PM

#

git sometimes fails for really big repos so you just gotta let it keep retrying

neon crane Sep 21, 2024, 6:10 PM

#

eh

#

flanterm and limine releases cant be that big

grand shadow Sep 21, 2024, 6:11 PM

#

the kernel youre using is also using some of the new iovec_iterator stuff but it should be fine too

#

for sockets only anyways

neon crane Sep 21, 2024, 6:12 PM

#

wtf

#

guess i gotta use deinit

#

oh finally its working

#

80 kib/s

grand shadow Sep 21, 2024, 6:14 PM

#

bruh wtf

neon crane Sep 21, 2024, 6:14 PM

#

now 25 kib/s

grand shadow Sep 21, 2024, 6:14 PM

#

when git cloning I usually get 700kb/s+

neon crane Sep 21, 2024, 6:14 PM

#

and back to 75

neon crane Sep 21, 2024, 6:14 PM

#

grand shadow when git cloning I usually get 700kb/s+

now i am using tcg

grand shadow Sep 21, 2024, 6:14 PM

#

ohhh

neon crane Sep 21, 2024, 6:14 PM

#

that might be a cause for slowdown

grand shadow Sep 21, 2024, 6:14 PM

#

that explains it

neon crane Sep 21, 2024, 6:14 PM

#

i can't use acceleration cause qemu on windows sucks

grand shadow Sep 21, 2024, 6:14 PM

#

oh

neon crane Sep 21, 2024, 6:15 PM

#

tbh i've never seen this lack of acceleration as a downside. it has made me actually optimize my shit

#

L

neon crane Sep 21, 2024, 6:15 PM

#

grand shadow oh

1 more thing

#

how do i restart without losing data

#

i assume you have some kind of caching

grand shadow Sep 21, 2024, 6:15 PM

#

sync

#

or

#

poweroff

#

since it calls into init to kill everyone and then sync

neon crane Sep 21, 2024, 6:16 PM

#

poweroff didn't work

grand shadow Sep 21, 2024, 6:16 PM

#

which is safer ig

#

wait what

neon crane Sep 21, 2024, 6:16 PM

#

gimmie a second

#

oh wtf

#

did it lock up

grand shadow Sep 21, 2024, 6:16 PM

#

interesting

obsidian pivot Sep 21, 2024, 6:20 PM

#

I downloaded the astral image

#

root.img and the iso

#

and keyboard input doesn't quite seem to work

#

in Xorg

neon crane Sep 21, 2024, 6:22 PM

#

#

😬

#

deadlock?

obsidian pivot Sep 21, 2024, 6:22 PM

#

grand shadow Sep 21, 2024, 6:22 PM

#

do infor registers

grand shadow Sep 21, 2024, 6:22 PM

#

obsidian pivot

wtf?

neon crane Sep 21, 2024, 6:22 PM

#

i cant

grand shadow Sep 21, 2024, 6:22 PM

#

what are you passing to qemu

neon crane Sep 21, 2024, 6:22 PM

#

grand shadow do infor registers

this ass console won't let me scroll back or page

obsidian pivot Sep 21, 2024, 6:22 PM

#

qemu-system-x86_64 -cdrom bootdisk.iso -M q35 -m 512M -enable-kvm -drive file=root.img,if=none,id=nvm,format=raw -device nvme,serial=deadbeef,drive=nvm -boot order=d -monitor stdio```

neon crane Sep 21, 2024, 6:22 PM

#

so you can only see the bottom half

#

which isnt too interesting

#

obsidian pivot Sep 21, 2024, 6:23 PM

#

-monitor stdio

#

to qemu

neon crane Sep 21, 2024, 6:23 PM

#

well i can't restart now because i fear that i can't replicate the bug!

grand shadow Sep 21, 2024, 6:23 PM

#

does qemu have instruction disassembly

neon crane Sep 21, 2024, 6:23 PM

#

yeah

grand shadow Sep 21, 2024, 6:24 PM

#

if its in the idle thread its a mutex deadlock or if its in some random context not halted its a spinlock deadlock

obsidian pivot Sep 21, 2024, 6:24 PM

#

obsidian pivot ``` qemu-system-x86_64 -cdrom bootdisk.iso -M q35 -m 512M -enable-kvm -drive fil...

it works using the console option

neon crane Sep 21, 2024, 6:24 PM

#

obsidian pivot Sep 21, 2024, 6:24 PM

#

I got it

#

nosmp worked

neon crane Sep 21, 2024, 6:24 PM

#

eip always says its in this runqueue thing

grand shadow Sep 21, 2024, 6:24 PM

#

does keyboard0 exist

obsidian pivot Sep 21, 2024, 6:24 PM

#

somehoq

neon crane Sep 21, 2024, 6:24 PM

#

so it's probably a deadlock with your runqueue

grand shadow Sep 21, 2024, 6:25 PM

#

obsidian pivot nosmp worked

that qemu cmdline doesnt even use smp tho lol

grand shadow Sep 21, 2024, 6:25 PM

#

neon crane so it's probably a deadlock with your runqueue

well shit

neon crane Sep 21, 2024, 6:25 PM

#

also i wasn't using smp

grand shadow Sep 21, 2024, 6:25 PM

#

its all being rewritten soon anyways meme

obsidian pivot Sep 21, 2024, 6:25 PM

#

grand shadow that qemu cmdline doesnt even use smp tho lol

yeah idk what's up with that

grand shadow Sep 21, 2024, 6:25 PM

#

I'll try building boron here later and see if I can replicate

#

gotta finish this iovec_iterator stuff for ext2

neon crane Sep 21, 2024, 6:26 PM

#

Also is there no top/htop/what

grand shadow Sep 21, 2024, 6:26 PM

#

nah

neon crane Sep 21, 2024, 6:26 PM

#

grand shadow Sep 21, 2024, 6:26 PM

#

that needs /proc

neon crane Sep 21, 2024, 6:26 PM

#

for god's sake that's why I pass -p to mkdir

grand shadow Sep 21, 2024, 6:26 PM

#

did mkdir -p not work?

neon crane Sep 21, 2024, 6:27 PM

#

its working now but it's strange to fail like that...

#

i guess it should be good enough for a restart right

grand shadow Sep 21, 2024, 6:30 PM

#

should be enough

#

weird that poweroff isnt accessible tho

#

I leave it in /usr/sbin/

neon crane Sep 21, 2024, 6:30 PM

#

is /boot linked to the boot drive

#

nvm uh

#

i guess a different question would be how would I access a drive other than the boot drive from limine

grand shadow Sep 21, 2024, 6:31 PM

#

mount -d device directory fs

neon crane Sep 21, 2024, 6:32 PM

#

from limine

#

caught these

#

ill just put them on an ISO normally and try to boot them

grand shadow Sep 21, 2024, 6:33 PM

#

yeah I boot from the iso and just mount the nvme disk as root

obsidian pivot Sep 21, 2024, 6:33 PM

#

what nics do astral support

neon crane Sep 21, 2024, 6:33 PM

#

grand shadow yeah I boot from the iso and just mount the nvme disk as root

ye

obsidian pivot Sep 21, 2024, 6:33 PM

#

I need to know what to pass to qemu

grand shadow Sep 21, 2024, 6:33 PM

#

obsidian pivot what nics do astral support

virtio only rn

obsidian pivot Sep 21, 2024, 6:33 PM

#

what cmd line would I pass?

grand shadow Sep 21, 2024, 6:33 PM

#

obsidian pivot I need to know what to pass to qemu

theres literally a cmdline with it

#

in the message you downloaded the isos from

obsidian pivot Sep 21, 2024, 6:33 PM

#

oh really

#

I just looked at the links

grand shadow Sep 21, 2024, 6:33 PM

#

-netdev user,id=net0 -device virtio-net,netdev=net0

obsidian pivot Sep 21, 2024, 6:34 PM

#

thanks

#

shit

grand shadow Sep 21, 2024, 6:34 PM

#

grand shadow I leave it in /usr/sbin/

my guess would be xterm's default path doesnt add sbin

obsidian pivot Sep 21, 2024, 6:34 PM

#

qemu-system-x86_64: -netdev user,id=net0: network backend 'user' is not compiled into this binary```

#

nooo

#

guess I'm removing my qemu binaries

#

so is there some sorta make uninstall

#

good

#

TIL don't use make install with a system prefix

neon crane Sep 21, 2024, 6:37 PM

#

so it is booting

#

but its so fucked up

#

it thinks KbdInitialize is an import

grand shadow Sep 21, 2024, 6:38 PM

#

damn

neon crane Sep 21, 2024, 6:38 PM

#

not supposed to!

grand shadow Sep 21, 2024, 6:38 PM

#

at least it starts meme

neon crane Sep 21, 2024, 6:38 PM

#

#

left: my host
right: astral

neon crane Sep 21, 2024, 6:38 PM

#

grand shadow at least it starts <:meme:575445094589661215>

yeah

#

honestly great job with it

#

hopefully it gets even better

#

the other objects seem to be fine, uh...

#

#

here's what it should have said!

obsidian pivot Sep 21, 2024, 6:40 PM

#

astral is cloning obos

neon crane Sep 21, 2024, 6:40 PM

#

to authenticate that it was built on astral yknow 😉

obsidian pivot Sep 21, 2024, 6:40 PM

#

does astral have cmake?

grand shadow Sep 21, 2024, 6:40 PM

#

neon crane

this is without the kyboard module?

grand shadow Sep 21, 2024, 6:40 PM

#

obsidian pivot does astral have cmake?

nope, only on the host

neon crane Sep 21, 2024, 6:40 PM

#

i transplanted my own but i could have just removed it lol

grand shadow Sep 21, 2024, 6:40 PM

#

for the build

#

oh lol

#

still good at only one thing seems to be broken

neon crane Sep 21, 2024, 6:41 PM

#

neon crane

anyways about that comment above

#

the memory manager is like the only thing blocking me right now from starting to write userspace code

#

i need to write the userspace facing parts

#

and shit

#

i wonder if i should just write a "first version" "by feels"

#

and then refactor later

obsidian pivot Sep 21, 2024, 6:42 PM

#

I'll use my host to generate build files on the image

#

then build in astral

#

and see how it goes

grand shadow Sep 21, 2024, 6:46 PM

#

neon crane the memory manager is like the only thing blocking me right now from starting to...

do you plan on doing a posix server or doing something more like nanoshell

neon crane Sep 21, 2024, 6:46 PM

#

not sure what you mean by "something more like nanoshell"

#

I want to have a posix-ish DLL you can import and use at some point

grand shadow Sep 21, 2024, 6:47 PM

#

neon crane not sure what you mean by "something more like nanoshell"

I mean like not unix

neon crane Sep 21, 2024, 6:47 PM

#

there will be some "native" stuff and you can additionally compile stuff for posix compatibility

grand shadow Sep 21, 2024, 6:47 PM

#

I see

neon crane Sep 21, 2024, 6:47 PM

#

tbh I might actually start with an mlibc powered ecosystem

#

like, boron.dll will expose system calls, mlibc will link with it

#

and applications will link with mlibc as a shared object

#

example:

int sys_vm_map(void *hint, size_t size, int prot, int flags, int fd,
               off_t offset, void **window)
{
    // N.B. hint ignored. Maybe not when I actually go implement it in boron?
    void* Address = NULL;
    BSTATUS Status = OSAllocateVirtualMemory(CURRENT_PROCESS_HANDLE, &Address, (size + 4095) / 4096, OS_VM_RESERVE, ConvertMmapProt(prot));
    if (FAILED(Status))
        return StatusToErrno(Status);

    if (fd) {
        // map the file in with another syscall
    } else {
        // actually commit it as anon memory
    }
    
    return 0;
}```

#

or even simpler

void sys_exit(int status) {
    OSExit(status);
    __builtin_unreachable();
}```

grand shadow Sep 21, 2024, 6:52 PM

#

that'd be cool to see a more NTlike OS run mlibc

neon crane Sep 21, 2024, 6:52 PM

#

about the cwd, i think that i'll have the current working directory as a concept in userspace and not in kernel space.

wild marsh Sep 21, 2024, 6:52 PM

#

neon crane example: ```c int sys_vm_map(void *hint, size_t size, int prot, int flags, int f...

the hint is needed for dynamic library loading

#

for MAP_FIXED

neon crane Sep 21, 2024, 6:52 PM

#

but i'm not sure how that'd work with e.g. get_cwd

neon crane Sep 21, 2024, 6:53 PM

#

wild marsh the hint is needed for dynamic library loading

why

#

can't you relocate the object if it can't be loaded at that addr

#

(side note, that reminds me a lot of the preferred base concept in PE)

wild marsh Sep 21, 2024, 6:53 PM

#

you need one mmap call for each phdr

neon crane Sep 21, 2024, 6:53 PM

#

yeah

#

and if you cant load the phdr at that address what do you do?

#

fail?

wild marsh Sep 21, 2024, 6:54 PM

#

yeah and they need to be placed at known offsets relative to eachother

neon crane Sep 21, 2024, 6:54 PM

#

or relocate

neon crane Sep 21, 2024, 6:54 PM

#

wild marsh yeah and they need to be placed at known offsets relative to eachother

oh that's fucked up

wild marsh Sep 21, 2024, 6:57 PM

#

in the best case the mlibc linker more or less does: ```
top_address = max(phdr.p_vaddr + phdr.p_memsz for each phdr)

spc = mmap(NULL, top_address, PROT_NONE, MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);

for each phdr:
mmap(spc + phdr.p_vaddr, phdr.p_filesz, protection_for(phdr), flags_for(phdr), elf_fd, phdr.p_offset)

neon crane Sep 21, 2024, 6:57 PM

#

ah so this could be encoded with a single OS_VM_RESERVE AllocateVirtualMemory and some file map calls

#

i'll think about it when the time comes though

wild marsh Sep 21, 2024, 6:58 PM

#

protection_for being self explanatory, flags_for deciding between MAP_SHARED/MAP_PRIVATE depending on whether it's ro/rw

wild marsh Sep 21, 2024, 6:59 PM

#

wild marsh in the best case the mlibc linker more or less does: ``` top_address = max(phdr....

it might also use a bump allocator starting at a fixed address instead of the initial mmap call, and/or it can use only anonymous mmap and read the contents of the file into the mappings

neon crane Sep 21, 2024, 6:59 PM

#

wild marsh in the best case the mlibc linker more or less does: ``` top_address = max(phdr....

ah, so no preferred base like i thought

#

but wait

obsidian pivot Sep 21, 2024, 6:59 PM

#

obsidian pivot then build in astral

turns out cmake doesn't work like that

wild marsh Sep 21, 2024, 6:59 PM

#

yeah elfs don't have a preferred base

neon crane Sep 21, 2024, 6:59 PM

#

-1?

#

wheres the size

wild marsh Sep 21, 2024, 6:59 PM

#

top_address is the size

#

assuming the base is 0

neon crane Sep 21, 2024, 6:59 PM

#

but it adds p_vaddr

#

what if the image isnt relocatable and has a nonnull vaddr

wild marsh Sep 21, 2024, 7:00 PM

#

then it's not a shared library

#

and it shouldn't have been processed in that function

neon crane Sep 21, 2024, 7:00 PM

#

i see

wild marsh Sep 21, 2024, 7:00 PM

#

actually no, my bad

#

we don't assume base=0

#

    if (mlibc::sys_vm_map(nullptr,
            highest_address - object->baseAddress, PROT_NONE,
            MAP_PRIVATE | MAP_ANONYMOUS, -1, 0, &mappedAddr)) {

neon crane Sep 21, 2024, 7:00 PM

#

i see

neon crane Sep 21, 2024, 7:01 PM

#

wild marsh ```c if (mlibc::sys_vm_map(nullptr, highest_address - object->ba...

PROT_NONE with MAP_PRIVATE | MAP_ANONYMOUS simply reserves the address range with no actual backing right?

wild marsh Sep 21, 2024, 7:01 PM

#

yeah

#

that's the idea anyway

neon crane Sep 21, 2024, 7:01 PM

#

i see

wild marsh Sep 21, 2024, 7:01 PM

#

we reserve as much as the library needs, then use MAP_FIXED to place chunks of the file in the correct spots

#

the actual code is in this function https://github.com/managarm/mlibc/blob/master/options/rtld/generic/linker.cpp#L418

neon crane Sep 21, 2024, 7:04 PM

#

oh that reminded me

#

mmap is kind of weird

#

it behaves in terms of pages

#

that means you can totally split a mmap range in half by doing
void* addr = mmap(pages: 6)
munmap(addr + 0x2000, pages: 2)

#

itll leave only the pages addr, addr+0x1000, addr+0x4000 and addr+0x5000 up

#

doesnt mean i can't adjust to that behavior it's just funny

wild marsh Sep 21, 2024, 7:06 PM

#

neon crane that means you can totally split a mmap range in half by doing void* addr = mmap...

yeah that behavior is kinda annoying

#

you can also cause a split with mprotect

#

making the middle have different permissions

neon crane Sep 21, 2024, 7:09 PM

#

wild marsh yeah that behavior is kinda annoying

if you do things naively it's actually very easy

#

cause you just zero out the relevant PTEs

#

but i'm... not seeking to do things too naively

#

like an mmap wont just reserve physical pages instantly

wild marsh Sep 21, 2024, 7:09 PM

#

wild marsh making the middle have different permissions

oh and you can cause a split by mmap with MAP_FIXED too

wild marsh Sep 21, 2024, 7:10 PM

#

neon crane like an mmap wont just reserve physical pages instantly

in managarm it was annoying since the address space is made up of mappings which are ranges of pages

#

so on an operation that split it i needed to replace the existing mapping with 3 new mappings instead

neon crane Sep 21, 2024, 7:12 PM

#

That's how I plan on doing things too, with something known as a VAD list (virtual address descriptor)

winged basin Sep 21, 2024, 7:14 PM

#

wild marsh in managarm it was annoying since the address space is made up of mappings which...

it's irking

#

you can tell mmap()'s origins

#

the APIs were added in 4.2BSD but not implemented, and the design was obviously intended for a page-by-page based representation

grand shadow Sep 21, 2024, 7:23 PM

#

alright, I think the whole fs interface now uses iovec_iterator to pass the buffer

#

I am keeping the old vfs_{read,write}() as a helper but all of the action is done on the new vfs_{read,write}_iovec()

neon crane Sep 21, 2024, 7:24 PM

#

what is iovec?

grand shadow Sep 21, 2024, 7:24 PM

#

the scatter gather list in posix for syscalls like readv

#

I decided to make all my I/O use it

#

and handle it using an iovec_iterator_t that keeps track of some important things and exposes an api to copy data to/from a buffer or a ringbuffer and even get pages from the iovecs

#

so that it can get treated like a normal buffer transparently

#

its still not upstream

#

my reason to do this originally was to support copying userspace buffers from system calls like readv or sendmsg which supply a list of iovecs

#Astral