#limine on arm progress

1 messages · Page 1 of 1 (latest)

weak panther
#

moving from the scal|ux thread, if i do the same "fix" as for kvm (not letting limine write to the terminal in vprint), it reaches the kernel, and the kernel can write stuff, but it's glitchy as you can see

20220809_212522.jpg
lament wren
#

yoo

weak panther
#

so at least the bug is shared between kvm and bare metal

#

which makes it potentially easier to deebug

lament wren
#

can you send me source code?

weak panther
#

i can rebase my local fixups and push to my fork ig

agile smelt
#

Anything for non-pi?

cobalt linden
#

this is not Pi specific

#

it's uefi

#

it's just that the only uefi arm64 hw qookie has is a pi4

weak panther
#

pi4

cobalt linden
#

tyoeo

#

tyepo

#

i give ip

weak panther
#

lmfao

chrome perch
#

try the up arrow

cobalt linden
#

neve tgonna give you up

cobalt linden
chrome perch
chrome perch
cobalt linden
#

still quote annoying

cobalt linden
#

it

lavish plume
weak panther
#

usb uart adapter

lavish plume
#

did you make it yourself

weak panther
#

no it's just a generic ftdi breakout board

lavish plume
weak panther
#

i bought it a long while ago from china

lavish plume
#

I google up "ftdi breakout board"?

cobalt linden
#

use another Pi as an adapter

#

if you build a single level shifrr

#

you can use an Arduino too

lavish plume
#

and I blew my arduino up

cobalt linden
#

if you want serial rx only you don't even need a level shifter

weak panther
lavish plume
#

perfect

#

just need some jumper wires

#

which I have by the handful already

weak panther
#

beware of one thing though, if you are on windows, the official ftdi driver will brick clone chips by overwriting their vendor and product id

lavish plume
#

how do I know if I have a clone chip (since I'm on windows rn)

weak panther
#

idr how it detects them but i remember hearing about it a while ago

keen vault
weak panther
#

i haven't, and i'm working on limine trunk which doesn't have stivale2 anymore

lavish plume
#

sadge

#

I always wondered why stivale2 had to go

weak panther
alpine scroll
#

Nvm i got it

gilded saffron
#

AMAXING!

weak panther
#

hopefully i got it right on the first try but i wouldn't be surprised if it just fucks up the pts completeely

#

and indeed, it doesn't work even in qemu tcg

#

it reaches the kernel and then fails to print anything

cobalt linden
#

aarch64 does so many aborts its a criminal according to federal us law now

weak panther
#

yeah but a synchronous external abort on an instruction fetch?

#

while still in limine (after switching pts in el1 i presume)

#

with tcg too

#

(can't do -d int with kvm, same as on x86)

#

ig i just accidentally tried to execute code from mmio

weak panther
#

okay that was a really dumb mistake

# 
uint64_t old_virt = virt & (old_page_size - 1);
``` vs ```c
uint64_t old_virt = virt & ~(old_page_size - 1);
#

lets see if this still works on real hw

#

lets see if this still works on real hw

#

the logic for it is pretty generic too

#

okay great, screen output is no longer messed up on my pi4

#

last thing that remains is seeing if smp works or not

#

since it used to be broken but maybe it got fixed along with everything else

#

okay so it doesn't die a horrible death, but the ap doesn't seem to make it into the kernel eithere

#

although it makes it far enough that limine says it succeeded

# 
smp: BSP MPIDR is 0x0
smp: Found candidate AP for bring-up. Interface no.: 0x1, MPIDR: 0x1
smp: Successfully brought up AP
smp: Found candidate AP for bring-up. Interface no.: 0x2, MPIDR: 0x2
smp: Successfully brought up AP
smp: Found candidate AP for bring-up. Interface no.: 0x3, MPIDR: 0x3
smp: Successfully brought up AP
cobalt linden
#

if this is in asm check your atomics

weak panther
#

the ap sets the boot flag correctly

#

otherwise limine would complain that it failed to start

#

but it doesn't seem to make it into the kernel

#

since i added ap startup to the test kernel, where the aps just print some text to the screen

#

inb4 it gets stuck in the __atomic_store_n

#

oh, seq_cst __atomic_store_n uses ll/sc and not stlr

#

so i imagine the ldar on the other side just doesn't notice it

#

hm no i am saying retarded things

#

yeah no it uses ldar/stlr too, ll/sc is for fetch_add

#

okay update, so the issue with limine not crashing in get_memmap anymore was because i wasn't calling it anymore :^)

#

so i still have that to debug

haughty gorge
#

bruv

weak panther
#

at least i know whereabout it's broken ig

keen vault
#

@weak panther I'd map the framebuffer using 0b00100010

#

where 0b0010 is WT R alloc transient

#

so it only takes a cacheline if you read from it

#

otherwise uncacheable

#

the proper way to map it is WB and flushing after each cacheline sized chunk tho ofc

#

but that's not part of the boot proto

haughty gorge
#

rip

#

@weak panther could you check if it works any better if you rebase

weak panther
#

what did you push

#

also i have a suspicion on whats going on

weak panther
#

my suspicion is that after setting SCTLR.M=0 (regardless of .C value since both mean uncached), the cpu stops looking into the dcache, because nothing can possibly be committed to it anymore

#

similar to how x86 cpus handle uncacheable memory that comes from mtrrs (as per intel's manual)

#

so the plan is to just flush the entire dcache before turning off the mmu before reclaiming uefi memory

weak panther
lament wren
#

that sounds fun

weak panther
#

honestly quite incredible ```armasm
mrs x0, clidr_el1
bfxil x0, x0, #3, #24
add x0, x0, x0 lsl #1

# 
x0 = clidr_el1
x0 = (x0 >> 24) & ((1 << 3) - 1)
x0 = x0 + x0 << 1
cobalt linden
#

bfm 😋

weak panther
#

quite risc indeed

keen vault
#

but yes I believe dcache is disabled by setting M=0

weak panther
#

well icache can't have data that's not in the dcache or ram

keen vault
#

I mean 🤷‍♂️

weak panther
#

it can be out of date but not ahead as in

keen vault
#

oh that's what you meant sure

weak panther
#

i am currently reverse engineering linux asm to figure out how it flushes the whole dcache :^)

#

it goes through all the cache levels, and if they're data or unified it computes the line size, max way and index and does an invalidate by set/way for each index and way combination

#

hmmmm this was dropped in a later patch, citing that it's impossible to actually do properly

#

hmmmmmm i wonder, what if i just create alternative page tables like the x86_64 code does

#

that seems easier frankly

chrome perch
#

turns out, arm always checks the d-cache, even if the pagetables and C bit say not to

those bits only control if a line-fill happens!

weak panther
#

huh

#

the behavior i'm observing (or at least i think i am observing) seems to suggest that after i turn the mmu (and caches by extension) off, reads of data that was in the cache but hasn't been written to ram yield data from ram and not cache

#

since i enter the function fine, i disable caches, i can call other functions after that point, but returning out of the function that disabled the caches fails with a jump to a bogus address

#

like ```c
void foo() {
sctlr.{m, c, i} = {0, 0, 0}

print("caches off");

} // crash

chrome perch
#

some of what i said may only apply to certain implementations

weak panther
#

i'm currently making it so that the code creates new page tables, so that it can keep caches enabled

#

and i have to create new page tables because the uefi ones are stored in uefi reclaimable memory, and when limine tries to reclaim it, it fucks them up

chrome perch
#

ah, ive not interacted with efi or aarch64 much yet

#

most of my work has been baremetal arm32

weak panther
#

since he didn't manage to figure that out

#

and like, this was enough to upset it even when running in qemu

chrome perch
#

one thing to note, is that qemu ignores all cache control flags, and everything is always coherent

weak panther
#

yeah, but it still emulates a tlb iirc

#

at least somewhat

chrome perch
#

more specifically, qemu just doesnt have any cache
mmio always traps and emulates
ram load/store benefits from the host cache

weak panther
#

and kvm emulates caches somewhat but not fully

#

so like disabling caches will break atomics, but wrong device memory type doesn't matter

weak panther
#

okay damn yeah, this just worked

#

now all that's left is figuring smp out

chrome perch
#

i also took a quick look over some uefi stuff, and it looks like uefi starts with either identity paging or no mmu?

#

and if your switching from identity->higherhalf, then you dont want all of the relocation patches to be applied to your kernel...

weak panther
#

afaicu, the uefi spec says you are entered with the mmu enabled, and the page size is guaranteed to be 4K (if i understood the spec right)

#

in this case i'm transitioning from the uefi identity mapping to my own identity mapping

#

because there's still some code to run between limine reclaiming uefi memory (incl uefi page tables) and limine entering the kernel

chrome perch
#

but you also need to call SetVirtualAddressMap() at some point, when you leave identity mode

weak panther
#

we're doing this long after ExitBootServices()

chrome perch
#

if you want the runtime services to survive

weak panther
#

x86 code doesn't seem to care about it so i'm not gonna either for now

#

or well, the page tables limine gives to the kernel have an identity mapping as well as a higher half mapping

#

whatever the kernel does then is up to it

chrome perch
#

i was thinking along the lines of just taking an existing kernel, and running it thru a pe32 linker rather then an elf linker, and generation relocations

#

and then skip the start.S stage, and just call main() directly

#

but if the entire kernel gets relocated by the firmware, to suit the identity mapping, you break yourself when going into higher-half mode

weak panther
#

you could go the gnuefi route, generate a pe32 header in assembly, implement the elf relocator yourself, then objcopy the elf into a flat binary

chrome perch
#

linux avoids the entire relocation mess, by using the MMU to relocate the code

weak panther
#

so uefi starts the gnuefi _start, which calls the elf relocator and then calls the elf entry

chrome perch
#

so at runtime, its at the addr the linker picked

#

LK does the same

weak panther
#

i've no clue how gnuefi relocates stuff (afaict it just parses _DYNAMIC?)

chrome perch
#

but from what ive seen, it seems like its always best to have an efi stub?

weak panther
#

well when compiled for uefi, limine is a full blown uefi program until it's time to enter the kernel

chrome perch
#

yeah, and thats where LK/linux differ, they are the kernel

weak panther
#

it does a bunch of last minute preparations, exits boot services, reclaims uefi memory then enters the kernel

chrome perch
#

i was thinking, could i make a single pe32 binary, that is both the uefi application AND the kernel?

weak panther
#

for booting linux from limine you can just do efi chainloading until i bother to implement the linux boot proto

chrome perch
#

but when the firmware loads that binary, it will apply relocations to make it runnable at some random load addr, in the identity mapping
and when i try to switch to higher-half mode, it will break

weak panther
#

hmm

#

ig you could try doing that by combining 2 executables

#

one lower half and one higher half

#

and transitioning from one to the other in the middle

chrome perch
#

and at that point, you can just make an efi stub

#

a tiny uefi application, that will ExitBootServices, map the real kernel at a higher addr, turn on the mmu, and jump over

weak panther
#

true

chrome perch
#

and the real kernel is linked differently, for that addr

#

each chunk has its own version of libc

#

but the stub could also just lack any print, which greatly reduces the deps

#

u-boot has a good example of that, one sec

#

and as a hint about what they are doing, this file even contains a memset and memcpy implementation

#

so its likely being linked fully stand-alone

#

but i'm also seeing a lot of signs of x86 in here

weak panther
#

yeah the code segment stuff is kinda odd

chrome perch
# 
 * U-Boot only works in 32-bit mode at present, so when booting from 64-bit
 * EFI we must first change to 32-bit mode. To do this we need to find the
 * correct code segment to use (an entry in the Global Descriptor Table).
#

i think this is a 64bit efi app, that then drops to 32bit mode to launch the real uboot?
and then uboot may go back to 64bit, depending on the final kernel?

weak panther
#

kinda similar to what limine does. limine runs in whatever mode uefi left it for the most part, except for when it's time to enter the kernel

#

when in 64-bit mode, it drops to 32-bit mode so that the code to enter the kernel can set up a well known state without being different on different bitnesses

chrome perch
#

what does uefi wind up doing, if it can potentially support 2 bit sizes?

#

can a 64bit uefi run a 32bit app?

weak panther
#

not sure, but limine supports being compiled for uefi x86_64 and uefi ia32 (and uefi aarch64 once it's done meme)

#

but on x86, the common kernel entry code is always run in 32-bit protected mode

#

even if the rest of limine is 64-bit

#

and that common code then sets up 64-bit long mode again, this time with everything well known

weak panther
#

which is why uefi-ia32 still supports 64-bit kernels (and checks if the cpu supports 64-bit mode at runtime)

chrome perch
#

my rough understanding, is that GetMemoryMap tells you the entire memory map
normal ram
runtime services ram?
and probably special mmio ranges?

SetVirtualAddressMap() is then used to tell the firmware what virtual address your moving the runtime services code to
and then you can create your own paging tables, and switch to them
and then run your kernel from wherever you want
and the firmware will automatically patch the set of function pointers for the runtime services?

weak panther
#

i don't really know how the runtime services work

#

as far as i'm aware, the limine boot protocol doesn't say anything about them

chrome perch
#

from what ive been able to gather, the runtime services is just a chunk of native code, that is declared in the memory map
when you switch to proper virtual mode, you must assign it a virtual address, update the map, and inform the firmware with SetVirtualAddressMap()

the firmware will then patch itself, to assume its now at that base addr
and you must now only call it via that addr

#

it will also patch the table of runtime service function pointers

#

and that then provides you with some basic functions:
get/set efi vars
reboot/shutdown
get/set time

weak panther
#

interesting

#

hmm, lets see if smp just happens to work now or if i need to debug it some more

chrome perch
# 
[root@system76:~]# efibootmgr -v
BootCurrent: 0004
Timeout: 1 seconds
BootOrder: 0004,0000,0001,0002,0003
Boot0000* ubuntu        VenHw(99e275e7-75a0-4b37-a2e6-c5385e6c00cb)
Boot0001* UEFI:CD/DVD Drive     BBS(129,,0x0)
Boot0002* UEFI:Removable Device BBS(130,,0x0)
Boot0003* UEFI:Network Device   BBS(131,,0x0)
Boot0004* UEFI OS       HD(1,GPT,27c99b08-455d-4dfe-a44f-6150cbc09ef8,0x800,0x100000)/File(\EFI\BOOT\BOOTX64.EFI)..BO
#

this linux tool will then have the kernel call into the runtime services, in this case to print the efi vars

#

but the tool can also modify them

#

BootCurrent tells you that the currently running os, was loaded by Boot0004

#

BootOrder is the order the firmware will try each option in

#

HD(1,GPT,27c99b08-455d-4dfe-a44f-6150cbc09ef8,0x800,0x100000)/File(\EFI\BOOT\BOOTX64.EFI)
this says to search a GPT disk for a given PARTUUID:

[root@system76:~]# blkid /dev/nvme0n1p1
/dev/nvme0n1p1: UUID="7DBC-2698" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="27c99b08-455d-4dfe-a44f-6150cbc09ef8"

and then run a specific path on that partition:
-rwxr-xr-x 1 root root 136K Mar 1 10:53 /boot/EFI/BOOT/BOOTX64.EFI

#

so in thoery, each OS can be its own entry in the efi vars, and you can change the boot order with the runtime services

#

and via BootCurrent, you know which one is yourself, so you can edit your own entry and not mess up others

weak panther
#

quite unfortunate, smp is still borked

#

the cores wakes up and enter the limine stub, but when the kernel then tries to gain control of them, they don't make it

chrome perch
#

did you setup the paging tables? and discard the L1's?

weak panther
#

the core seems to manage to enable the mmu, since if it failed, core 0 would time out waiting for it to set a flag and panic

#

i'll look into it later, it's nearly 4 am and i should go to bed

chrome perch
#

heh, i always do that

weak panther
#

so, for tomorrow what's left: debug smp and rebase onto limine trunk

#

and it should be good to submit as an initial support pr

#

dtb support and linux boot proto can come at a later time

chrome perch
#

arm linux is easy

weak panther
#

yeah its not difficult, apart from the fact that it wants to be placed as close to start of ram as possible

#

and having to patch the dtb to set cmdline and initrd pointers

chrome perch
#

thats easy, and i have example code

#

255-270 is probably the most complex part, filling in the memory map and ranges

weak panther
#

i could probably make it so that linux boot proto is only supported if uefi gives us the dtb (otherwise just go use efi chainloading)

#

and inb4, you can get acpi in place of a dtb

chrome perch
#

the pi4 uefi can give you both acpi and dtb

weak panther
#

yeah its configurable, but the arm systemready sr specification explicitly prohibits dtbs

chrome perch
#

another thing, is that every time i bring up efi infront of the open source guys, they freak out and say its not needed 😛

#

the hard-core ones, that complain about all of the bloat in tianocore and having even a single byte not backed by source 😛

agile smelt
#

What gets me is the remaining blobs needed to boot a rockchip system
Looks like there's quite a bit of magic ram timing tools and magic firmware formats that are undocumented

#

Which is fine if uboot somehow wraps that in a gpl compatible way or whatever, but what about tianocore

#

I found some uefi port for this board I have, but no uboot source, so 🤷‍♂️

chrome perch
haughty gorge
#

since limine's default pagemap preserves the identity map, you can just call the runtime services via the system table

#

including the call to relocate to a new pagemap

keen vault
#

I just pass it on as bootloader reclaimable

weak panther
#

yeah limine reclaims it and passes it as available