Serra

[small field]

An x86 bootloader that I've been working on over the past couple of months.
Still under construction, and licensed under the MIT license <3
https://github.com/NunoLealF/Serra/

[small field]

Here's the roadmap as of now

[reef shale]

coole

[small field]

Right now it isn't anything that special
But that's because I'm still "laying out the foundations" and initializing different parts of the system

[small field]

Here's a demo using int 13h
This is.. very inefficient to say the least ||(each one of those calls turns off protected mode, initializes a 16-bit real mode GDT/IVT, writes one pixel, and then returns back to protected mode)||, but it's only meant to be a demonstration of what it can do

Right now I have about 1,500-3,000 lines of code (depending on whether you count comments/blanks)

[reef shale]

thats a lot

[small field]

That's more comments than all of Limine btw, but with 5-10% of the actual code size lmao

[small field]

https://github.com/NunoLealF/Serra/blob/main/Boot/Legacy/Rm/Rm.asm
(here's the code that handles this, for reference)

[small field]

My goal after all of this is done is for it to be a boot manager like Clover or Ventoy, but more modern/feature-rich
Maybe kind of like macOS's Recovery mode? (Has a partition manager, a browser, etc.)

[mighty quartz]

i would argue that having almost 1 comment per line of code is actually excessive

without knowing more i expect idiocy such as x86asm ; write 69 to register AX mov ax, 69

ok i took a look at the code

it's way overcommented

while too little comments is bad, too many comments is also bad

like it's pretty much the meme i posted above

[civic marsh]

Generally I find that comments which describes what something does are super redundant.. with comments in asm being an exception

Like if the code ain't readable enough to need comments which describes what it does.. just fix the code

[mighty quartz]

also comments need constant upkeeping

it is all to common to read some codebase and its comments are outdated because the code was changed but the comments weren't

and overcommenting (like in this case) makes the codebase super cluttered

like there is no need to write in a comment half the SDM... just put a paragraph number referencing whichever manual and move on

[civic marsh]

Yup

I usually only leave comments if I do something unexpected or there is some important side effect

[fiery loom]

# This moves the decimal value '69' in base 10 to the least significant byte of AX, also known as: register AL, usually found in x86 CPUs
move al, 69

[small field]

I can't really disagree with you, it is pretty excessive lol

I only did it like that to make it easier to understand/remember in the future however
It also helps you keep track of things

[small field]

lol

[small field]

Believe it or not I used to be even worse lmao

[civic marsh]

I mean thats wild but id argue this is better

describing what a function does is not bad, especially if its at an API boundary

[small field]

Fair
I can't pick many examples as that project has like, 400 lines of code

But I commented like that everywhere

[manic star]

"usually found in x86 cpus" where else would it be found

[fiery loom]

someone could create a cpu and name it the same :B

[small field]

Finally finished implementing exceptions
I just need to clean up my code and add comments now

[small field]

All ISRs and regular IRQs are supported, here's an example

[reef shale]

so is Serra a bootloader or kernel?

[small field]

My goal is for it to be a boot manager / sort of like a mini OS

So.. it's a little bit of both
It's hard to describe

[small field]

601 lines of C excluding comments
1650 lines of C including comments

https://tenor.com/view/gigachad-chad-gif-20773266

[small field]

Also, I changed it to give actual progress reports

[solid harbor]

wait

you have more comments than code

that's a bit concerning

especially at 600-1600 slocs :^)

[small field]

I am a master at overcommenting

[reef shale]

lmao

you should maybe comment less

more productive

[glad wren]

as i already said somewhere - good code comments itself

if you need more comments than code... that says something

😬

[small field]

It still does that, I just write way too many comments

Most of the code is understandable without them
But if I don't include them, people can find it hard to understand (not because of how it's structured, OSDev is just complicated)
My goal is for people to easily be able to read my code, which is why I do that

[glad wren]

my point being you shouldn't need comments if you write good code

because anyone reading the code won't need the comments to understand it

[civic marsh]

[vital night]

Which is part of the reason my VFS code needs more comments

[small field]

I will start working on CPUID now

Nothing too complicated though

[small field]

yay

[ripe zephyr]

aweome

[vital night]

Don't bother with checking for cpuid support. It most likely does and the ID bit in eflags is never guaranteed to be set if cpuid is supported

[small field]

I don't want to crash the system if it isn't though

It's easy to check though so I don't care too much ¯_(ツ)_/¯

[vital night]

If you want an example of cpuid existing but the ID bit not being set, try VBOX

[small field]

Isn't it that the ID bit can be modified, but not that it's set by default?

[vital night]

Just test for it's existance by running it and seeing if you #UD

[small field]

On QEMU it's clear by default but it lets me modify it

[small field]

That was my initial plan but my exception handler has no way to return from an #UD

[vital night]

Skill issue

anyway I gtg

[small field]

Prior to using the CPUID instruction, you should also make sure the processor supports it by testing the 'ID' bit (0x200000) in eflags. This bit is modifiable only when the CPUID instruction is supported. For systems that don't support CPUID, changing the 'ID' bit will have no effect.

[small field]

Managed to implement a really basic printf()
I'm surprised it was so simple honestly

I put it off for a long time as I thought it was going to be really hard
But it ended up taking me less than an hour haha

[small field]

Seems to work fine

[small field]

I've been on a hiatus because of school, but I've finished implementing cleaning up the e820 memory map

(Here's qemu's memory map for reference)

[small field]

This was around 300 lines of code in Bootloader.c alone

So much code

The next step is to work on supporting FAT16/FAT32, and to load files
After that, I can start working on 'proper' drivers for things like USB, NVMe, etc.

I made an interface that allows me to call real mode functions from protected mode, so int 13h should be accessible.
It could be slow though

[small field]

I haven't started on actually interpreting the filesystem, but I did change up a few things in my bootsector - the only thing really left to do is to parse the MBR table(s) to figure out which LBA I should load (right now I'm just assuming the bootloader/bootsector starts at LBA 0, but that's not safe)
I don't intend on making a MBR for now, just a VBR, so that's probably going to be necessary

[small field]

I did push up the location of the 2nd stage bootloader to avoid interfering with MBR/GPT though
Instead of starting immediately after the bootsector, it now starts at LBA 64

[small field]

Nvm, I forgot the hidden sectors field existed; now FAT16/FAT32 should be more simple (I hope)

[small field]

I cleaned up some of the code
I'm starting to run out of space, and I think I'm thinking about changing the 1st/2nd/3rd stages completely

[small field]

Right now, my code loads the 2nd stage in the area from 7E00h-CE00h in memory
But not only is that pretty little, I also keep adding code that isn't really necessary; for example, almost a full blown terminal implementation, IDT/PIC code when I don't need it yet, E820 memory map code, etc.

[small field]

My idea was to shift most of that onto the 3rd stage, and to create a minimal 2nd stage that just loads the 3rd stage, checks for errors, etc.

That way, I could still keep pretty much everything, but reserve more space for the parts that need it (in the 3rd stage), and also keep it relatively simple

[small field]

I ended up doing this
Second stage is in Legacy/Init, third stage is in Legacy/Core

I've gotten almost everything in the second stage to work - a proper GDT, code that interfaces with real mode, proper memory functions (memset/memcpy/whatever), etc.

With the way I've laid out my second stage, I only have about 16 sectors / 8KiB to work with, and I've already used up about 4KiB
That leaves me with about 4,000 bytes for:

• Reading the disk using real mode interrupts (could be hard)
• Reading and interpreting a FAT16/FAT32 file system
• Loading the third-stage bootloader (hopefully as a file on the root directory, like bootx64.efi would be)

[small field]

I've started working on this again, I'm actually really close to my goal here!
EDD/int 13h extension support is now available, and I've started reading off of the BPB (from now on, everything will work inside of a FAT16/FAT32 filesystem)

[small field]

Here's the symbol table (this is only for the second stage bootloader, most of what I implemented in the third stage bootloader isn't even here)
About 20 functions in only 6KiB, give or take

For reference, here's the symbol table of the (old) third-stage bootloader:

For now, this looks incomplete, but I'm actually pretty close to loading the next stage

[small field]

After a lot of trial and error (12 hours of work lol), I've actually gotten a proper FAT16 driver working

Works with FAT32 as well, just as I expected
It also handles different sector sizes just fine on FAT16, though I am having some issues on FAT32 (seems like the number of clusters isn't changing, weirdly enough)

I'm still happy though

[tribal trout]

Awesome job man

[small field]

Thanks <3

[small field]

Turns out, mformat is updating this field with the number of 512-byte sectors for some reason, not the number of logical sectors

[small field]

Still need to iron out a few problems though
Namely, I.. still don't really understand how the layout for files and folders work, and my driver isn't giving me answers that actually make sense

[small field]

(For example, I'm trying to search for a file called Boot/Core.bin, but no matter what cluster number I actually specify, it still finds Core.bin)

This is with a completely random cluster number, and yet it still finds the file in cluster 3..?

Wait nvm I just realized I forgot to add the cluster offset

[small field]

Okay, awesome, not only does it work but it fixed the bug I was having with FAT32 only being able to read from the root directory

[small field]

Looks like it can successfully read files as well:

(Output of xxd cat.mp4)

00000000: 0000 0020 6674 7970 6973 6f6d 0000 0200  ... ftypisom....
00000010: 6973 6f6d 6973 6f32 6176 6331 6d70 3431  isomiso2avc1mp41
00000020: 0000 0963 6d6f 6f76 0000 006c 6d76 6864  ...cmoov...lmvhd
00000030: 0000 0000 0000 0000 0000 0000 0000 03e8  ................
00000040: 0000 11b6 0001 0000 0100 0000 0000 0000  ................
00000050: 0000 0000 0001 0000 0000 0000 0000 0000  ................
00000060: 0000 0000 0001 0000 0000 0000 0000 0000  ................
00000070: 0000 0000 4000 0000 0000 0000 0000 0000  ....@...........
00000080: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000090: 0000 0002 0000 0894 7472 616b 0000 005c  ........trak...\
000000a0: 746b 6864 0000 0003 0000 0000 0000 0000  tkhd............
000000b0: 0000 0001 0000 0000 0000 11b6 0000 0000  ................
000000c0: 0000 0000 0000 0000 0000 0000 0001 0000  ................
000000d0: 0000 0000 0000 0000 0000 0000 0001 0000  ................
000000e0: 0000 0000 0000 0000 0000 0000 4000 0000  ............@...
000000f0: 011a 0000 01f2 0000 0000 0024 6564 7473  ...........$edts

[small field]

For reference here's cat.mp4

[grizzled raven]

hey man i got a question which languages do i need to learn before doing anything close to this or even just booting in general?

[small field]

(Low-level) assembly and C should be all you need
It's a little difficult to do this from protected-mode, but I'm pretty sure you can easily come up with something in real mode

[grizzled raven]

all types of C? (c++,C#,etc) or a specific one

[small field]

Just C should be enough
But some basic Assembly is also pretty important

[grizzled raven]

thanks appreciate your response! was stuck on where to start prior.

[small field]

No problem! Good luck with your endeavours

[small field]

After a short break, I'm back on track
I'm happy to report that it can actually read files from disk, on both FAT16 and FAT32

Now it can load the 3rd stage bootloader
The only thing left now is to finish commenting my code, and then implement that

[small field]

https://github.com/NunoLealF/Serra/blob/main/Boot/NewLegacy/Init/Disk/Disk.c
This took me 3,000 years but I finished it

[small field]

Finally

It's not hard to just have dd hardcode something
But it's reading and jumping to an actual file in a FAT16/32 filesystem

It works flawlessly on FAT16 and FAT32 with varying sector sizes

[small field]

Better (this shows a normal message from the third stage bootloader)

I also added a toggle to hide all non-important messages (anything but warnings/fails/errors/the message in this screenshot)

Starting to get long

[manic star]

ever heard of uefi

[small field]

My plan is to implement efi support later

[small field]

Added IDT/PIC support to the third stage; this was pretty easy as I only had to copy from my old bootloader and tweak a few things
Right now, my roadmap is:

• Organize and comment my code, review the documentation I have, etc.
• Finish bringing over other features from my old bootloader (for example, E820 is missing)
• Finally, work on implementing more features (PCI, ACPI, VESA..)

My 'old' bootloader is also Serra, it's just that I'm rewriting it
I already had E820 support in my 'old' bootloader, but I haven't copied it over to the rewritten/'new' part

[small field]

also, yes, IRQs work

[small field]

Some updates:

• I've added full support for E820
• I have like 5,000 lines of code at this point

[small field]

For now, I want to focus on

• getting basic CPUID support
• implementing PCI (and services like ATA PIO, and NVMe support)
• disk/filesystem support
• getting VESA and EDID info
• if it isn't too complicated, basic ACPI support
• transferring into long mode, and loading the next part of the bootloader (way out in the future)

[small field]

I'm writing a boot manager, not just a regular bootloader, so basically just an OS with the ability to load other operating systems from a menu; that's why there's a next part

My idea is to have something like this:

• BIOS bootsector -> second stage BIOS bootloader -> third stage BIOS bootloader (you are here) -> kernel (the user can load their operating system from here)
• UEFI loader -> kernel (the user can load their operating system from here)

[mighty quartz]

good luck honestly

writing a boot manager/bootloader isn't at all a trivial endeavour

[small field]

Thanks, I appreciate it

I have about a month until classes start, so I'll see if I can get most of that done until then
Realistically speaking, probably not, but

[mighty quartz]

it may distract you from your actual OS though

given it's a project in and of itself

[small field]

The other one on my github? That one's been abandoned for a while, I only got around to writing a tiny part of the bootloader haha

[mighty quartz]

uh no? i mean this one

[small field]

Oh lol

[small field]

Added some (very basic) CPUID support; everything seems to be going fine, even on different CPUs
(Screenshots are from Pentium III (-cpu pentium3), AMD EPYC (-cpu EPYC) and Hygon Dhyana (-cpu Dhyana) respectively)

I didn't even know what Hygon CPUs were until now

Also works with KVM

[small field]

This isn't CPUID-related, but just for fun, here's some proof that it works with only 1MiB of memory (-m 1)

[small field]

Revised roadmap:

• getting basic CPUID support
• basic ACPI support (Nothing that complicated, just finding the RSDT/XSDT and passing it onto the next stage of the bootloader if possible)
• basic disk drivers (int13h, and maybe ATA PIO, but nothing else; I really don't want to deal with PCI)
• basic FAT16/FAT32 drivers (I already implemented this last time)
• getting VESA and EDID info
• anything else that requires real mode
• jumping to the actual boot manager itself (with paging and long mode)

[small field]

Added a function to read the RSDT a few days ago

(screenshots are from VMware and QEMU respectively)

[small field]

I'm too tired to work on anything else though

fdhjpogfdjh

[small field]

Added some basic VBE support; nothing complicated, and I still need to flesh it out (for now I've only implemented ax=4F00h), but it works

[small field]

Comparison between QEMU (SeaBIOS), VirtualBox and VMWare

(QEMU has VBE 3.0, VirtualBox has VBE 2.3 and VMware has VBE 2.2; I'm surprised QEMU is the only one to support VBE3 haha)

[small field]

Proof that VBE definitely works

(This isn't an actual window or anything, it's just manually displaying a raw 16-bit image)

[small field]

Added basic EDID support, along with more VBE functions

Sadly Virtualbox and VMWare both don't support EDID (or, at least, int 10h, ax 4F15h, bl 01h is not supported)

[small field]

I've finally finished implementing proper VESA/EDID support
(Screenshots are with VESA+EDID on, VESA on and EDID off, and VESA+EDID off)

[small field]

Added some basic SMBIOS and FAT support (the FAT driver is the same as before so it was quite easy)
And I went ahead and added a (very simple) memory allocator

[small field]

It's been a few months, but I finally made a simple demo (that just shows a logo and a PS/2 mouse)
(My PS/2 driver is kind of broken, but eeehhhhhh)

[small field]

Yeah the PS/2 mouse driver is really broken, it doesn't even work on virtualbox (which is explicitly emulating an actual PS/2 mouse lol)

[small field]

Working on a diagram that shows everything

[tribal trout]

oh you're working on this again, nice

[small field]

ty ty

I totally revamped the readme https://github.com/NunoLealF/Serra

[small field]

I also finished this

This usually wouldn't be necessary, but this is my senior project, so I have to come up with this even if it's not that necessary; I also have a report I need to fill out later on, I'll need to present this a few months from now, etc. etc.

[civic marsh]

is this also why u have more comments than code

lol

[small field]

Somewhat yeah

[civic marsh]

fair enough

[small field]

Right now my goal is to work on a proper paging implementation (for once) and go to long mode
And then I can start working on actually making it a proper boot manager

[small field]

(I'll probably just identity-map all memory below 1MiB and all reserved/non-free memory in my e820 mmap, and then map everything else to like 0xE000000000000000)
(That, or I could just map the kernel, and then the kernel can identity-map/allocate/do whatever as needed)

(I say kernel because it's supposed to be like a tiny OS that runs at boot, not just a bootloader lol)

[civic marsh]

I mean you probably want to provide a direct map

like limine does

identity mapping is unecessary

[civic marsh]

preferably in the higher half as that is often reserved for the kernel

otherwise people have to relocate it

[lime hemlock]

0xE000000000000000 is not a valid address

[small field]

I may be stupid

(I meant 0xE00000000000 lol, my bad)

[lime hemlock]

also not valid

[small field]

I mean yeah it's 0xFFFFE00000000000 (bit 47 repeats)

It's just more confusing when you put it that way, since 0x0000600000000000 -> 0x0000700000000000 -> 0xFFFF800000000000 -> 0xFFFF900000000000..
(assuming 5-level paging isn't enabled)

[small field]

I actually managed to switch into long mode, wow

(It isn't anywhere near complete - in fact it literally only identity-maps the first 2mib and nothing else - but it does mean that my IA-32e page tables work / that my paging implementation isn't borked, so I'm happy)

[small field]

Seems to work on Virtualbox too

[small field]

A few updates:

• In order to simplify things, I decided to make the project 64-bit only
• • For now, the "bootloader" part (Boot/Legacy/) will be in IA-32e mode still, but the "kernel" (Common/) should be 64-bit
• I've also finalized my paging strategy, and I should actually be done with that quite soon
• • I've been spending the past few weeks contemplating where to map the kernel, what sections should be identity-mapped, etc., and I'm finally done with planning that
• uintptr and intptr types have been added to Stdint.h

Identity-mapping is already done, I just need to finish mapping all free memory regions to one consecutive region ||(the 256th PML4, or FFFF__80__0000000000h-FFFF__80__7FFFFFFFFFh - I just call this 80h.low)||, and then I should be able to enable paging and load the kernel

This isn't much ive just been procrastinating a lot

[civic marsh]

Serra is a kernel now?

[small field]

The idea is that it's a boot manager, that kind of functions as its own tiny operating system - it should eventually load other OSs, but the idea is to jump into a small kernel to let the user interact with the system (a little like GRUB or Ventoy) until the user picks something

So, for example:

• The bootloader part would load the Serra kernel
• In turn, the kernel would go through the available filesystems / try to find OSs it can boot to, and then give the user a menu (as well as a few other amenities, like a shell)
• Once the user picks an OS to boot, it would load it (unloading pretty much everything else in the process, like paging), before transferring control

[civic marsh]

Aight

wouldnt it be more convenient to build all that into the bootloader itself

or is the kernel just like a stage 2

[small field]

Kinda, yeah

[civic marsh]

fair enough

im curious on how u are going to do disk stuff in 64bit

[small field]

I should always have int 13h or EFI services as a fallback, but I do want to eventually implement PCI/SATA/NVMe/etc. drivers in the kernel

(right now /Boot/Efi is totally empty, but the idea is that the kernel should be universally compatible with both EFI and legacy BIOS - in theory there shouldn't be any one hardcoded method of accessing the disk)

[civic marsh]

Ah yeah, how much extra work is it returning from long mode

my bootloader is written for 32bit specifically so I can return quite easily

[small field]

Probably quite a fair bit lol
But I'm not in long mode yet, I'm still working on paging rn

[civic marsh]

fair

[small field]

This was IA-32e mode (which is technically long mode? I think), but it wasn't a proper paging setup in any way, I was just trying to see if my functions for making page tables worked lol
(It only mapped the first 2mib of memory and nothing else, and it made no effort to deal with 64-bit interrupts, so triggering one would just crash the entire system)

[small field]

I do have a SaveState() and RestoreState() function in my bootloader that's supposed to simplify that

/* void SaveState(), RestoreState()

   Inputs: (none)
   Outputs: (none)

   These two functions are called by Shared/Rm/RmWrapper.c, and they basically just save and
   restore the state of the system (for example, they might restore the IDT, paging, etc.)

   (TODO: actually comment this better, the specifics of what this is guaranteed to vary
   quite a bit)

*/

descriptorTable IdtDescriptor;

void SaveState(void) {

  __asm__ __volatile__ ("cli");

  return;

}

void RestoreState(void) {

  __asm__ __volatile__ ("sti");
  LoadIdt(&IdtDescriptor);

  return;

}

[small field]

My idea for the paging structure is:

• The first PML4 ||(0000000000000000h-0000007FFFFFFFFFh; I just call this 00h.low)|| is identity-mapped using 2MiB pages for compatibility reasons; this doesn't have too much overhead
• The 256th PML4 ||(FFFF800000000000h-FFFF807FFFFFFFFFh; I just call this 80h.low)|| is dedicated to usable memory (not taken up by page tables, reserved memory, etc.); in theory, this should just be one contiguous virtual address space that can be used by anything, even if the actual memory map is split into a thousand different sections
• The last PML4 ||(FFFFFF8000000000h-FFFFFFFFFFFFFFFFh; I just call this FFh.high)|| is dedicated to the kernel itself; in theory, this should be remapped from the usable memory region (80h.low)

[small field]

Paging is mostly done, all that's left before I transfer control to my 64-bit kernel is to actually read the program/section headers, and map them accordingly - everything else is already done

Makefile has been rewritten to show each section more clearly, and to operate out of the root folder instead of /Boot/Legacy

(here's the full structure, for reference)

There's also been a couple changes to the readme, though they are from a few months ago

[small field]

In the meantime, I've already started working on the kernel.. or, at least, transferring control to it
There's a specific .pmstub section (which corresponds to https://github.com/NunoLealF/Serra/blob/main/Common/Kernel/Stub.asm) in the final EFI file that:

• enables paging (with a given pml4)
• switches to long mode
• transfers control to the kernel entrypoint (FFFFFFFF80000000h)

There's also 1 MiB of stack space below that last PML4 entry (more specifically, FFFFFFFF00000000h~FFFFFFFF7FFFFFFFh is reserved for the stack, and FFFFFFFF80000000h-FFFFFFFFFFFFFFFFh for the kernel itself)

[small field]

The infotables thing is also important - the whole reason why the bootloader part of this has taken so long is that I want everything else to be ready by the time I switch into the kernel
That means the bootloader is doing more than just loading the kernel; for example, it scans for video modes (and automatically switches to the best one / provides a framebuffer to the kernel if possible), ACPI tables, etc.
It genuinely wouldn't take that much time to adapt this to something like the multiboot protocol (though with some limitations); it's closer to a mini OS than a regular bootloader, even if the aim is to be a boot manager

(really the only question is how to deal with drivers/additional files - I still have no idea where exactly I'll load those or how they'll interact with the kernel)

[small field]

Okay looks like the long mode kernel is working just fine, it even works if I pass a string to it from protected mode

[small field]

It's been a few days, and

• A good chunk of the comments/documentation has been rewritten
• Long mode and paging works flawlessly, and I've added PAT support (and made the framebuffer write-combining)
• I've fixed a few bugs in the disk (EDD/int13h), filesystem (FAT) and graphics (VBE) drivers, as well as in some of my paging code
• • This also means that, from my experience, it actually works flawlessly on real hardware (!); YMMV though
• Paging/identity-mapping is also faster (I used to just blindly identity-map the first 512 GiB; now I just focus on what's needed, and it's a lot faster)
• In theory, my disk/FAT drivers are now a lot more efficient, and more resistant to faults
• • (It doesn't even have a problem with wacky configurations like 27 sectors per cluster lol)

[small field]

All that's left is to review (and fix any bugs / rewrite documentation / etc.) a few hundred lines of code, and then I can ditch protected mode entirely

In theory, once I implement a proper memory manager and filesystem driver, I should be able to branch out a lot more
Right now my roadmap is basically

• Come up with a proper plan for the transition process (what info should be given over to the kernel / what environment it should start in / etc.)
• Enable basic features; SSE/SSE2, VBE graphics modes, etc.
• Create a half-decent memory manager
• • Once I can do that things should be a lot easier
• Create half-decent ACPI and PCI drivers
• • This should allow me to load the rest of the information I need, and deal with things like AHCI/NVMe/USB

[small field]

I've finally documented the rest of my bootloader code, and I did two things:

• Pass on more information to the kernel via an info table (before this, I just passed a string that I used to test with)
• • This was a little challenging, especially since I had to create a table with 64-bit pointers that was usable from 32-bit code ||(I ended up just using a uint64~void* union, where the 32-bit side uses uint64 and the 64-bit side can use either)||, but it does seem to work; very much a work in progress though
• Switch to the C23 standard
• • Bools no longer need to be defined, __attribute__((noreturn)) has been replaced by [[noreturn]], I can declare fixed-width enums, I can use [[maybe_unused]] on non-essential variables, and most importantly, #embed lets me import data directly into the source code, which is really useful for fonts and such

(For reference, here's the current version of the info table - it's obviously still really incomplete and kind of a work in progress, there are many things that need to be done, but my goal is to eventually have the kernel use all of this for the hardware interface layer)

(As well as the code that sets it up, in Boot/Legacy/Core/Bootloader.c)

[small field]

Switching to C23 does severely reduce compiler compatibility (the only compilers that work from my testing are GCC 13 with -std=c2x and GCC 14+ with -std=c23), but it really is useful

[small field]

I made a few bugfixes, and added x87/SSE/SSE2 support
Works on real hardware

(this is a graphics test pattern thing, with -mmmx -msse -msse2 -O3, from 64-bit mode)

[tribal trout]

At least your thing works on real hardware, my kernel instantly triple faults on everything I’ve ran it on

[small field]

oof, sorry to hear about that

[small field]

Kernel is now position-independent, yay

[small field]

After like 8 hours of work I've finally managed to come up with my own (very incomplete but functional) homegrown EFI library

This is nowhere near the kernel so far, and there's genuinely a lot of work left, but im happy with the result

[small field]

For reference here's the code that prints that (the 12345 is just to test out the timer function, it goes 1 -> 12 -> 123 -> ...)

efiStatus efiAbi Bootloader([[maybe_unused]] efiHandle ImageHandle, efiSystemTable* SystemTable) {

  // Print something - we temporarily set the attribute (color code pretty much,
  // works *just* like VGA) to 0x0F instead of 0x07

  SystemTable->ConOut->ClearScreen(SystemTable->ConOut);

  SystemTable->ConOut->SetAttribute(SystemTable->ConOut, 0x0F);
  SystemTable->ConOut->OutputString(SystemTable->ConOut, u"Hi sigmas <3 \n\r");
  SystemTable->ConOut->SetAttribute(SystemTable->ConOut, 0x0B);

  // Wait for 5 seconds, printing the number each second

  char16* String = u"1";

  for (int i = 0; i < 5; i++) {

    SystemTable->ConOut->OutputString(SystemTable->ConOut, String);
    String[0]++;

    SystemTable->BootServices->RaiseTpl(TplHighLevel);
    SystemTable->BootServices->Stall(1000000);
    SystemTable->BootServices->RestoreTpl(TplApplication);

  }

  // Go to the next line, set attribute back to normal and then return.

  SystemTable->ConOut->SetAttribute(SystemTable->ConOut, 0x07);
  SystemTable->ConOut->OutputString(SystemTable->ConOut, u"\n\r");

  return EfiSuccess;

}

[small field]

I did try to keep mostly everything standard, although with a few changes (for instance, pretty much everything is camelCase or PascalCase, not SCREAMING_SNAKE_CASE)

Most services/tables/protocols aren't available (and won't unless I implement them in the future), and there is no support for anything but x64
But that's fine since this isn't meant to be anything like GNU-EFI or EDK-II, the idea is to just get things up and running for the kernel later on, and not much else

[small field]

Went ahead and cleaned up the codebase as well (both in terms of making sure everything worked together well / was properly commented and named, but also fixing any bugs (I used a few static analyzers))

Also changed the name of all Bootloader() functions to be distinct
(so, instead of having three identically named functions in Boot/Legacy/Init/Bootloader.c, Boot/Legacy/Core/Bootloader.c and Boot/Efi/Bootloader.c, it now has S2Bootloader(), S3Bootloader() and SEfiBootloader())

(S2 = second-stage (BIOS); S3 = third-stage (BIOS); SEfi = UEFI)

[small field]

Updated READMEs; much more informative now

[small field]

Now works with the graphics functions from the BIOS bootloader - pretty easy to port over, and now I can use essentially the same message system

(for reference this is what it looks like if the debug flag is turned off; the BIOS bootloader also has a similar function)

Also added functions to check the validity and presence of the system table, boot services, etc.
As well as ConIn and ConOut; in systems that don't have a console (or that aren't in emulated text mode), this sort of message reporting is automatically disabled to avoid causing crashes

[small field]

Appears to work flawlessly on real hardware as well, except for the color palette for some reason

White (0x0F) and light magenta (0x0D) text is still fine, but aqua/light cyan (0x0B) becomes white, dark gray (0x08) becomes black, etc.
Only a few PCs do this but it's still weird, I've never seen that with VGA text mode

(Maybe I need to disable the cursor blink?? lol)

[tribal trout]

you need to pay an extra subscription for those colors

[small field]

lol

[small field]

GOP implementation is working

(small correction, it's probably 24-bit not 32-bit)

[finite minnow]

No, it's almost certainly 32 bpp ARGB.

Have you sorted the issue with booting on apple? Haven't you tried my loader?

[small field]

GOP only exposes red, green, blue and reserved; I believe it has to be 32-bits anyway, but what i'm measuring here is how many red+green+blue bits are being used (the color depth)

Apple is still broken, but i've gotten a little closer to solving the problem at least (now it doesn't crash, it just returns to the OS, which indicates there's something wrong with their EFI implementation, or that it's below UEFI 2.0, or ConIn doesn't work at all)
I forgot about your loader though, i wasn't home when i got the notification so I couldn't test anything, let me try

[small field]

Okay so, it's not going to boot by default since Macs can only boot from /Efi/Boot/bootx64.efi or /System/Library/CoreServices/boot.efi (there is no EFI shell)
But on copying /Efi/Upptech to /Efi/Boot and renaming /Efi/Boot/antload.efi to /Efi/Boot/bootx64.efi, it does seem to boot in QEMU
I'll unplug my usb and try now
@finite minnow

[finite minnow]

what i'm measuring here is how many red+green+blue bits are being used (the color depth)
It's a physical framebuffer parameter. It should be reported via GOP.PixelFormat. In most cases, it's the same as GOP's one, 32 bpp ARGB. But sometimes, it's r5g6b5.

I'll unplug my usb and try now
would be awesome. ^_^ thanks.

[small field]

It gets stuck on the loading screen, nothing more

[finite minnow]

does it print some error code? any print? logo picture? That version just halts after jumping into kernel and printing some greeting and memory info. If it is like that, it has worked.

[small field]

Just the girl picture and a moving loading bar that doesnt really do anything else, i get it with qemu but it lasts ~0.1s so its probably stuck there

[finite minnow]

Is that on macbook real hw? 😄 the girl pic is a logo, if it reached there, it means it works. it rapidly disappears because the kernel erases the boot screen and prints its Trace screen. Can you photograph it?

[small field]

On QEMU it quickly disappears and then goes into a purple terminal thing
On my iMac, it's just stuck on the logo and loading screen, it doesn't progress further than that

Still impressive for a mac though, those are really hard to get right

[finite minnow]

My loader uses LocateProtocol() and a lot of other boot services, so if macbook painted my logo, it means, LocateProtocol() works there and your issue is in something else. So you say, it doesn't reach the "purple" terminal on real hw and gets stuck on the "girl" logo? hmmm.

It might be an issue, that I don't make fb memory write combined on x64 (since so far, it worked without that). I'll change that.

[small field]

My issue isnt with GOP to be clear, Macs just seem to have kind of a broken EFI implementation in general

More specifically something here causes it to return, so its something wrong with the system/boot services/runtime services tables

  // (Check the EFI System Table)

  if (SystemTable == NULL) {
    return EfiInvalidParameter;
  } else if (SystemTable->Hdr.Signature != efiSystemTableSignature) {
    return EfiInvalidParameter;
  } else if (SystemTable->Hdr.Size < sizeof(efiSystemTable)) {
    return EfiInvalidParameter;
  } else if (SystemTable->Hdr.Revision < calculateRevision(2, 0)) {
    return EfiUnsupported; // We don't support anything below UEFI 2.0
  }

  EfiInfoTable.SystemTable.Ptr = SystemTable;
  gST = SystemTable;

  // (Check the EFI Boot Services table)

  if (SystemTable->BootServices == NULL) {
    return EfiInvalidParameter;
  } else if (SystemTable->BootServices->Hdr.Signature != efiBootServicesSignature) {
    return EfiInvalidParameter;
  } else if (SystemTable->BootServices->Hdr.Size < sizeof(efiBootServices)) {
    return EfiInvalidParameter;
  }

  EfiInfoTable.BootServices.Ptr = SystemTable->BootServices;
  gBS = gST->BootServices;

  // (Check the EFI Runtime Services table)

  if (SystemTable->RuntimeServices == NULL) {
    return EfiInvalidParameter;
  } else if (SystemTable->RuntimeServices->Hdr.Signature != efiRuntimeServicesSignature) {
    return EfiInvalidParameter;
  } else if (SystemTable->RuntimeServices->Hdr.Size < sizeof(efiRuntimeServices)) {
    return EfiInvalidParameter;
  }

I'll test by removing the UEFI 2.0 requirement

[finite minnow]

I see, in an attempt to figure this, tell me please yet once, when you said, that on macbook, " it's just stuck on the logo and loading screen, it doesn't progress further than that", did you mean apple's logo or my loader logo? Because if the latter, everything is fine with system table and BS and RT tables, since my loader gets it from the parameter passed by fw.

[small field]

Yeah its that

Even though it supports GOP it doesnt report itself as reporting at least UEFI 2.0, never change apple
(I assume text mode is disabled as well but thats generally fine)

[small field]

The latter

[finite minnow]

so it's UEFI 2.0 but doesn't want to report as such and your check fails?

[small field]

Yeah

From what ive heard its more of a mix of EFI 1.1 and UEFI 2.0, it could just be reporting 1.1 for compatibility reasons

[finite minnow]

that's fun. So, you can move forwards now. And I should probably make fb write combined.

Or, maybe, if it really disables text output protocol,ConOut, it's impossible to see the error message from my loader. it prints it to ConOut. for example, if the hw doesn't support NX feature. as always crappified firmware makes it all screwed and apple is the 1st one. :[ Thank you for testing!

[small field]

I checked, and it works fine if I set it to check for revision 1.1 instead of 2.0

It still 'freezes' the menu but the moment I hit a key it goes back to normal / returns to the OS, so its clearly just transferring control of the framebuffer until it leaves the EFI application

(Still doesnt display anything, but thats fine since I'll set up GOP later on; at the very least I can confirm that it loads it correctly and that EFI_SIMPLE_TEXT_INPUT_PROTOCOL works flawlessly)

[small field]

Or, maybe, if it really disables text output protocol,ConOut, it's impossible to see the error message from my loader
Probably that

[finite minnow]

If you don't mind, we could check as it is, without adding anything. My loader if can't proceed, it prints the error code and asks to either hit any key to return to fw or hit s to shutdown. If this hang is due to such an error, when you hit any key (not <Enter>), it should get back to the UEFI menu. alternatively, if you hit s it should shutdown the machine (it uses runtime rs->ResetSystem(EfiResetShutdown)).

[finite minnow]

Yet one thing, @small field when you tested on macbook, did you put the Lybid directory to the USB stick root dir as well? without this it wouldn't work. I'm trying to understand why it didn't work. If you put both directories (efi and Lybid) to the stick root dir, then the only thing I could think of why it didn't work is apple disabled NX feature in their macbooks. even in 2013.

[small field]

I did, but honestly dont worry about it, macs are pretty weird

If I had to guess it probably has to do with some missing efi feature, i tested it on a 2010 iMac

[small field]

I'm going to leave it at this for today

[small field]

I found another pc that does the same thing, and it turns out the colors are still there, they're just really faint

Compare with Tianocore

[small field]

Updated the color scheme as well (on both BIOS and EFI loaders)

[small field]

I won't be home until tomorrow, i can't really test it

Sorry man, i wish you the best though

[small field]

Updated color scheme for [Info] messages, and implemented GetMemoryMap()
Not much left at this point

(Legacy loader also has the new color scheme, to be consistent)

[small field]

After like 10 hours of work, I've successfully

• Revamped the build system (you can now manage almost everything from a configuration file, makefile.config
• • You can now selectively build for specific platforms - only BIOS or UEFI, or only unpartitioned instead of MBR, etc.
• • You can also set the debug flag (which controls which messages are shown on the screen)
• Added full MBR support, as well as unpartitioned

[small field]

For reference, here's the (relevant part of) makefile.config:

# ------------------------------ Build options ------------------------------

  # Should the Debug flag be enabled? (false/true)
    Debug := true

  # Build for legacy (BIOS) targets? (false/true)
    BuildBios := true

  # Build for EFI targets? (false/true)
    BuildEfi := true

  # Build an unpartitioned, MBR or GPT image? (unpart/mbr/gpt)
    ImageType := mbr

  # What should the size of the FAT partition be, in MiB? (%d)
    PartitionSize := 24

  # How many sectors per FAT cluster? (%d)
    ClusterSize := 8

  # *If building a partitioned image*, then:
    # How large should it be, in MiB? (%d)
      ImageSize := 32
    # What sector size should be used, in bytes? (%d)
      SectorSize := 512
    # What LBA should the partition start at? (%d)
      PartitionLba := 2048

Comparison between Debug := false and Debug := true (BIOS)

Comparison between Debug := false and Debug := true (EFI)

[small field]

sfdisk -d output of the MBR image (ImageType = mbr, PartitionSize = 24, ImageSize = 32, SectorSize = 512, PartitionLba = 2048):

label: dos
label-id: 0x679ba5a8
device: Serra.img
unit: sectors
sector-size: 512

Serra.img1 : start=        2048, size=       49152, type=ef, bootable

sfdisk -d output of the unpartitioned image (ImageType = unpart, PartitionSize = 24, SectorSize = 512):

label: dos
label-id: 0x00000000
device: Serra.img
unit: sectors
sector-size: 512

Serra.img4 : start=           0, size=       65536, type=0

As far as I can tell it works fine on real hardware as well (I haven't tested it thoroughly however)

[small field]

Two updates:

• I added my own custom MBR, instead of relying on mbr-install to add it for me
• • This is a lot faster than the old method, but I'm still in the process of testing it
• 5,000 lines of code!!!

[lime hemlock]

damn what are you doing that you have 3.4k lines of comments

or ig rather an almost 1:1:1 ratio of blanks to comments to code

[small field]

I comment a lot

(or more importantly: this is technically something that counts towards my grade, so i wanted to make the code readable / easy to understand for people without an OSDev background)

[lime hemlock]

ah i see

[wary pecan]

there is such thing as excessive commenting

[small field]

Yeah, definitely
Though it's gotten better over time, i used to comment a lot more exhaustively

You can see the difference in the code i'm editing now lol

[wary pecan]

it is actually quite hard to comment code for me as someone who thinks in patterns and images rather than words

i am definitely in the "not really commented enough" part of the spectrum

i actually struggle to grasp how some people comment so much lol

[small field]

For me it's the opposite, if I don't overcomment my code it feels totally incomprehensible to me
(In general I have a lot of trouble understanding code where I can't make out what it's doing - I tend to skim code, not really read it, which is admittedly a bad habit tbf)

¯_(ツ)_/¯

[small field]

Okay, MBR is done!! At least it works super reliably on QEMU/Bochs

Like, here: it correctly boots /dev/sdc2, which is the only partition that has a valid bootsector (with the AA55h signature at the end), despite /dev/sdc3 also being marked as bootable
/dev/sdc1 is not bootable and it's automatically skipped, and even messing around with the order doesn't change anything

I can change the partition size, move it around, etc. and it works just fine

(I wasn't able to test CHS boot, but it's hard to find a machine that won't be fine with LBAs / extended int 13h)

[tribal trout]

dont bother

[small field]

Yeah my bootloader doesn't, only my MBR does lol

[small field]

GPT support is done as well, and it seems to work on the machines I've tested
To recap:

• Serra can now be built as an unpartitioned FAT image (unpart), an MBR-partitioned image (mbr), and a GPT-partitioned image (gpt)
• unpart images work fine with both BIOS and EFI, and have worked on all real hardware I've tested them on
• mbr images work fine with both BIOS and EFI, and have worked on most* real hardware I've tested them on
• • ||*One CSM system I tested it on just doesn't seem to recognize the MBR at all, for some reason; I think it might be looking for a BPB, since install-mbr also didn't work, and that uses a totally different MBR from mine.||
• gpt images only work with EFI, and have worked on all real hardware I've tested it on*
• • ||*Haven't been able to test it on everything yet; for example I have no idea if it works on a Mac||

Also rewrote some of the comments on the makefile, much cleaner now

Next things to do are to finish the EFI bootloader, which is what I was doing previously.
It still needs to look for ACPI/SMBIOS info, process the memory map, enable certain protocols, etc. before it can actually load the kernel

[tribal trout]

that's the most commented makefile ive ever seen lmao

[small field]

This is a pretty small update but my EFI bootloader can successfully find acpi/smbios tables
Works on real hardware as far as I can tell, but this also isnt that complicated

Not much left until I can load the kernel at this point. I just need to

• Properly set up a graphics mode (easy)
• Set up the memory map
• Locate the kernel file (easy)
• • And also add ELF loading capability, but I can just copy that from my BIOS loader
• Set up any necessary protocols

[small field]

Tbh it has to be, it's really hard to read otherwise

The ; \ after everything in an if statement, how only some statements aren't silenced with @, etc. etc.
Makefiles are really bad with shell script syntax, and within that context comments can really help with readability.

Comments like this help explain exactly what's going on - maybe the bootsector part is obvious, but Boot/Legacy/Init/Init.bin and Boot/Legacy/Shared/Rm/Rm.bin aren't exactly clear if you're new to the project
(Plus, it helps explain the overall 'flow')

There's also one more benefit, which is that some tools - like mformat - aren't super well documented, and while it's not hard to figure it out yourself, having a comment explaining exactly what's going on also saves time for the reader

[tribal trout]

fair

[small field]

• Kernel memory map now has a standard format (this also led to some speed improvements in the BIOS loader)
• EFI loader now dynamically allocates space for the kernel and leaves a defined amount of space for the firmware as well
• • (It also frees that space when exiting the EFI application)

[small field]

Not much left to do at this point - maybe CPUID?
But that can be done within the kernel so it's not even necessary, tbh

[small field]

I have yet to test this properly on real hardware, but this now

• Goes through all filesystem protocol handles (and then picks one that has the kernel in it)
• Reads a file named \BOOT\SERRA\KERNEL.ELF (the kernel!), dynamically allocating pages as needed for it
• Allocates space for the kernel stack

When I finish testing and commenting this, I should genuinely be really close; I hope to be able to finish the EFI loader by Friday at least

[small field]

After fixing a bug where it wouldn't actually close the file protocols on a 2011 iMac for some reason, it works perfectly fine on the real hardware I have

[small field]

The EFI loader now successfully loads the kernel/core

Comparison of EFI (/Boot/Efi) and BIOS (/Boot/Legacy) boot, with Debug enabled

Comparison of EFI (/Boot/Efi) and BIOS (/Boot/Legacy) boot, with Debug disabled

[storm fable]

looks so beautiful 😍

also higher half?

[small field]

As much as I want it to be (and it actually used to), it has to be identity-mapped and position-independent everything because that's the only way to keep it compatible with EFI without exiting boot services

[small field]

And thank you!!! I really appreciate it <3

[small field]

Also, just to be clear, graphics mode is totally supported, it's just disabled until I have a proper graphics stack working
Here is it displaying a bitmapped image on QEMU/Bochs

[reef shale]

Don't tell me you're trying to make an OS that doesn't exit the boot services... lol

[small field]

It's a boot manager that sort of functions like an OS
I do agree that you should always exit boot services, but that would make it nearly impossible to chainload EFI applications, for example

[tribal trout]

maybe copy the kernel from the identity mapped region to the higher half when you're about to jump to the kernel?

[small field]

To be clear, the kernel is just what I call the core of the boot manager, it's not the actual kernel you'd load from a boot manager
(I know that's confusing, but just think of it as a tiny operating system that eventually transfers control to an actual kernel)

So, it can't be higher half either, for the same reason
(Theoretically it could if it had no way to boot any EFI application whatsoever - like with BIOS boot - but the idea is that the kernel should be the same across both BIOS and UEFI (although with platform-specific handlers))

[tribal trout]

Ah

[small field]

On the topic of that, i've been working on a new format for the 'info table' (which is basically kind of like a boot protocol for the kernel itself - basically, a list of everything that's been collected by the bootloader)

All I have to do is to finish it, and then I should be able to start working on my kernel for once; in theory this should be mostly platform-agnostic
https://github.com/NunoLealF/Serra/blob/main/Common/InfoTableFormat.md

This has already been a thing for a while - and both the BIOS and EFI bootloaders do already deal with most of this - I'm just reworking the format

It is a really good representation of what info the bootloader collects, though

[small field]

I amended my kernel transition stub assembly thing to be able to take a return value
Same return value on BIOS and EFI (see the [Info] nya -> ... line), so you know it's booting the same code

This isn't a super significant milestone or anything I'm just happy I figured out how to switch back to compatibility mode since it's so poorly documented

[small field]

Also I've hit 6,000 lines of code
(This will probably go down tomorrow when I replace the old infotable (right now there's two of them defined) but yay)

[small field]

Just 11 days later and I've added

• 2577 new lines
• • 949 new blanks
• • 580 new comments
• • 1048 new lines of code

[small field]

Okay, i'm done with this now
I'm actually really happy with this, I can output graphics using the same code on both BIOS and UEFI machines
(As well as seamlessly switch between 32- and 64-bit mode)

[small field]

BIOS boot

-# First screen is drawn by the kernel using VBE framebuffer, second screen is drawn to B8000h after leaving the kernel (it waits for a few seconds before exiting with return code 2:1h)

EFI boot

-# First screen is drawn by the kernel using GOP framebuffer, second screen is drawn with ConOut after leaving the kernel - in EFI's case it just waits for userinput via ConIn before returning 2:0h

The first gradient is drawn by the same code on both systems:

      for (uint64 y = 0; y < InfoTable->Display.Graphics.LimitY; y++) {

        for (uint64 x = 0; x < (InfoTable->Display.Graphics.LimitX * InfoTable->Display.Graphics.Bits.PerPixel / 8); x++) {

          *(uint8*)(InfoTable->Display.Graphics.Framebuffer.Address + (InfoTable->Display.Graphics.Pitch * y) + x) = (y * 255 / InfoTable->Display.Graphics.LimitY);

        }

      }

There's a lot more shared than just the framebuffer, this is just a demo

Glad to see it working though

[small field]

There's a few things I need to work out - namely VBE barely seems to work on real hardware for some reason (it doesn't crash it just doesn't display anything either, works fine when I switch back to text mode)

It's also not a universal thing because it works on some hardware; really the only weird thing is that the two machines they don't work on are both from like 2011

[small field]

A few updates:

• Ring 3 support has been deprecated from EFI
• The ELF executable loader is now a lot better, and less prone to bugs - previously I was just directly transferring control to the beginning of the .text section without even making an attempt to deal with other sections, which obviously isn't correct
• • Now, both BIOS and EFI loaders explicitly read the program headers, map out the executable in memory (copying/setting data using SSE2)
• • They also check for alignment problems, and the .text section can be placed anywhere now
• I've further expanded C23 support
• • C2x (Boot/Efi/) and C23 (/Boot/Legacy, /Common) code now uses static_assert() to verify that types are defined correctly (since I use my own stdint.h), as well as verifying if __attribute__((packed)) is defined - this should sort out issues with non-GCC compilers
• • C2x and C23 code now uses alignas(n) instead of __attribute__((aligned(n))) to guarantee alignment
• • C23 code now uses auto when appropriate, which improves readability (especially in for-loops and such) - obviously this only applies to code where the type is already inferred

For example, here's a for loop from Boot/Legacy/Stage3/Bootloader.c (compiled with std=c23):

  for (auto Position = 0; Position < NumMmapEntries; Position++) {

    auto Start = Mmap[Position].Base;
    auto Size = Mmap[Position].Limit;
    auto End = (Start + Size);

    Message(Info, "Found type %d memory area from %x:%xh to %x:%xh (%d KiB)", (Mmap[Position].Type),
            (uint32)(Start >> 32), (uint32)Start, (uint32)(End >> 32), (uint32)End, (uint32)(Size / 1024));

  }

[small field]

Here's an example of the compile-time checks added to Common/Kernel/Stdint.h:

  // [Compile-time checks, and static assertions]

  #ifdef __has_attribute

    #if !(__has_attribute(packed))
      #error "Your compiler must support __attribute__((packed))"
    #endif

  #else

    #error "Your compiler must support GCC-style attributes (__attribute__(...))"

  #endif

  static_assert((sizeof(bool) == 1), "`bool` is not 8 bits.");

  static_assert((sizeof(int8) == 1), "`int8` has incorrect size.");
  static_assert((sizeof(int16) == 2), "`int16` has incorrect size.");
  static_assert((sizeof(int32) == 4), "`int32` has incorrect size.");
  static_assert((sizeof(int64) == 8), "`int64` has incorrect size.");

  static_assert((sizeof(uint8) == 1), "`uint8` has incorrect size.");
  static_assert((sizeof(uint16) == 2), "`uint16` has incorrect size.");
  static_assert((sizeof(uint32) == 4), "`uint32` has incorrect size.");
  static_assert((sizeof(uint64) == 8), "`uint64` has incorrect size.");

  static_assert((sizeof(intptr) == 8), "`intptr` has incorrect size.");
  static_assert((sizeof(uintptr) == 8), "`uintptr` has incorrect size.");

This is also quite important, since it's meant to be the same size on both 32- and 64-bit platforms

  typedef union __uptr {

    uint64 Address;
    void* Pointer;

  } uptr;

  static_assert((sizeof(uptr) == 8), "`uptr` is not 8 bytes");

[small field]

I've also added a makefile option (in makefile.config) to toggle graphics mode on or off:

  # Should graphics mode be used? (false/true)
    Graphical := true

(This works across both BIOS and EFI platforms, disabling VBE or GOP respectively)

For example, here's a comparison of EFI boot with Graphical := false and Graphical := true

As well as BIOS boot with Graphical := false and Graphical := true

[small field]

The kernel entrypoint now verifies the bootloader-provided info table (very exhaustively, ~300-400 lines of code) for any discrepancies

In order:

• It checks whether the pointer to the table is a null pointer
• It checks whether the table has the right signature
• It checks whether the table has the right version
• It checks whether the table has the right size
• It checks whether the table checksum is valid
• It checks whether the indicated architecture is supported (limited to x64)
• It checks whether the given CPU values appear to be valid (specifically, the table-provided CR0 must not be empty)
• If the table indicates that ACPI is supported, it checks whether the RSDP pointer is a null pointer
• If the table indicates that SMBIOS is supported, it checks whether the pointer to that data is null
• It checks whether the indicated firmware type is supported (limited to BIOS and EFI)
• If the firmware type is BIOS..
• • It checks whether the architecture is x86/x64 or not
• • It checks whether the method used to enable A20 is valid (unknown is valid, but it can't be some random value)
• • It checks whether the pointer to the E820 memory map is null
• • It checks whether the E820 memory map is empty
• • If PAT is supported, it checks whether the prior value is invalid (equals 0)
• • If PCI-BIOS extensions are supported, it checks whether the pointer to that data is null
• • If VBE is supported and enabled, it checks whether the mode number is zero, and if the pointer to any of the tables (InfoBlock, ModeInfo) are null
• If the firmware type is EFI..
• • It checks whether the provided image handle and system table are null
• • It checks whether the pointer to the EFI memory map is null
• • It checks whether the EFI memory map is empty
• • It checks whether the EFI memory map entry size is invalid
• • If GOP is supported and enabled, it checks whether the pointer to the protocol is null
• It checks if the stack size and pointer are both 4 KiB-aligned
• It checks whether the stack size is non-zero, and whether the pointer to the top of the stack is null
• It checks whether the kernel's executable type is valid/supported
• It checks whether the pointer to the kernel's executable header is null
• It checks whether the pointer to the kernel entrypoint is null
• It checks if the pointer to "usable" memory map (only contains free, non-overlapping entries, and in a firmware-agnostic format) is null
• It checks whether the "usable" memory map is empty
• It checks whether the display/graphics type is invalid (no display is okay, but it can't exceed the maximum)
• If EDID is supported..
• • It checks whether the preferred resolution is invalid (equals 0)
• • It checks whether the pointer to the EDID timing tables is null
• If a text mode (VGA or EFI Text) is being used..
• • It checks whether the text format is valid (ASCII and UTF-16 are both fine, it just can't be unknown or be outside of the valid set of values)
• • It checks whether the given console is at least 80 by 25 characters
• If a graphics mode (VBE or GOP) is being used..
• • It checks whether the pointer to the framebuffer is null
• • It checks whether the resolution is at least 640 by 480 pixels
• • It checks whether the number of bits per pixel is between 16 and 64, and divisible by 8
• • It checks whether the pitch / bytes per scanline is probably invalid (by checking if Pitch is less than (LimitX * Bpp / 8))
• • It checks whether any of the red, green or blue masks overlap each other
• • It checks whether the popcount of all of the masks together exceeds the number of bits per pixel
• It checks whether the disk access method is valid (must be int 13h, or EFI sfs)
• If the disk access method is int 13h..
• • It checks whether the drive number is probably invalid (not 00h-0Fh, 80-8Fh)
• • It checks whether the pointer to the EDD data is valid (if supported)
• If the disk access method is EFI sfs..
• • It checks if the handle list, protocol, etc. are null

[small field]

I also introduced a status code system that, on both BIOS and EFI systems, provides a pretty decent idea of what went wrong (it's not true or false, there's close to 30 different error messages it can display)
This should help immensely with debugging
-# EFI only deals with char16* so I have to manually convert every character, but instead of storing two separate tables I just do it on the fly

For example, this is what ends up getting shown if no errors are found (return value = entrypointSuccess = 0:0), on BIOS and EFI respectively:

But here's what shows up if I misalign the stack by doing KernelStackTop--:

Or if I change the drive number to 40h (which afaik never occurs on real hardware):

Or even if I just pass in a completely random address ||(within the range of usable memory)|| as the info table
Or change an element after calculating the checksum (which would happen if the table got corrupted midway, somehow)

[small field]

I'm now at 7,000 lines of code!!

Some updates:

• The kernel is starting to be fleshed out more; I can call EFI functions, set up CPU registers, etc. just fine, and there's a much more defined folder structure
• I'm linking it as an actual PIE executable now (up until now I've just been using -static-pie but that's broken)
• The kernel has been redesigned somewhat to be more architecture-agnostic (obviously it still isn't there, but everything x86 is wrapped up in its own layer for now)
• I've managed to add full SIMD support, all the way from SSE to AVX512f
• • I don't have any AVX512 hardware so I haven't been able to test it at all, but up to AVX2 everything appears to work pretty much flawlessly

[small field]

(The idea behind the SIMD stuff is to be able to optimize memory functions (like Memcpy/Memset) as much as possible)

[small field]

This took a while, but 1,000 lines of code later, I now have a reasonably well optimized Memcpy/Memset for

• ERMS systems (fast rep movsb/rep stosb)
• AVX-512f systems (tested, works fine!)
• AVX systems
• SSE2 systems
  All of which are coded in Assembly

AVX-512f does work, here's _Memset_Avx512f() in action on KVM QEMU:

(I can also confirm that it works reasonably fast with the framebuffer on most systems, but honestly it depends - my work laptop has an 11th gen i5 and the framebuffer is an order of magnitude slower than my PC with a 1 GHz embedded CPU from 2011, but my 4th gen i5 MacBook runs it incredibly fast)

QEMU tends to be pretty fast, Bochs is slightly slower but still more than good enough

[small field]

Yay bezier curve

(I think i'll use truetype for the console font - a bitmap one would undoubtedly be better but most of the formats out there are not that well documented and/or annoying to work with)

[small field]

I can now read PSF1 and PSF2 fonts
(to be fair this is just a demo but it does show that i can read it well at least)

BIOS and UEFI machine side by side

[small field]

I can draw boxes now
(I'm working on a simple graphics subsystem)

Works with other fonts (with a width under 8px at least)

[small field]

This took me an unreasonable amount of time, but I now have a fully working Print() function that supports both VGA and EFI text modes out of the box, along with graphics modes

It's very performant (with the exception of scrolling on graphics mode, since I don't have double buffering set up yet), and also seamlessly integrates with the bootloader text mode console

Print() comparison between VGA (80x25), VGA (80x50), EFI and Graphical (VBE/GOP)

Scroll test comparison between VGA (80x25), VGA (80x50), EFI and Graphical (VBE/GOP)

[small field]

I did a couple of tests, and from my observations ||(EFI text mode not included because it's just a function call)||...

• Text printing is actually pretty fast in graphics mode, and extremely fast in VGA text mode
• Scrolling can vary a lot in graphics mode, but is still very fast in VGA text mode
• • I did try to optimize it by precomputing how much I needed to scroll, so the amount of time it takes to scroll one line is the maximum amount of time it takes to scroll any amount of lines
• • Some hardware is so slow it takes one or two seconds to scroll, but most hardware can scroll 2-3 times per second, and some emulators can scroll upwards of a dozen times per second

But it's still surprisingly good, especially given the lack of double buffering - bootup times are extremely fast too

[small field]

Using QEMU's -icount 10,align=on option (which only allows one instruction to execute every 2¹⁰ nanoseconds, or just under a million IPS):

Bootup times

• Serra (w/debug messages disabled) takes 1-2 seconds to boot
• Serra (w/debug messages enabled) takes 5-6 seconds to boot
• Limine takes 25-30 seconds to boot
• GRUB2 takes 85-90 seconds to boot

(For reference that's supposedly 2x slower than a 386)

Even on Bochs (which is much more realistic), it boots up an order of magnitude faster than Limine or GRUB (<0.5s vs 2-5s)
Which makes sense due to the disk read speed, I'm just glad it's not Free95

[small field]

Oh my god how is Free95 THAT slow

[wary pecan]

ngl who cares about Free95 lol

also, i bet Flanterm is still faster than you are

[small field]

Oh no I know it's bad I just thought it was comically slow

[wary pecan]

i can tell

that's pretty nuts

[small field]

Hmm, maybe, let me test it

Wait oh it's a terminal emulator i'm stupid

[wary pecan]

i mean i thought we were comparing terminal emulators, so

it's what Limine uses anyways

[small field]

Yeah you're right, it's just that I can't really drop it into qemu lol
Still though, the code seems pretty good (and certainly a lot more complete!)

[wary pecan]

i like racing other terminals

it's fun

[small field]

In my case, the optimizations I'm using are

• -O3 and -mmmx -msse -msse2, always
• Having several optimized versions of memcpy/memset, and picking the right one
• • (More specifically: rep movsb/rep stosb for anything under 2048 bytes or any ERMS system at all, and AVX-512 + AVX + SSE2 implementations on top of that)
• Initializing a RGB color lookup table at boot, so I can easily convert RGB color values to framebuffer color values, no matter the type
• • Also, using [[reproducible]] on the function that does those conversions (yay C23), which is useful because of memoization
• Drawing entire lines at once, and only sending them over to the framebuffer when necessary
• • SIMD really really helps here for non-ERMS systems
    Which is actually pretty fast as long as you ignore scrolling

[wary pecan]

i think Flanterm can beat it without any of those optimisations

aka Limine basically

[small field]

Probably; just looking through the code there's a lot of lookup tables and such

(In this case I rely exclusively on a simple bitmap font and having no Unicode support whatsoever, which is going to be faster.. but also less limited)

[small field]

If anything the main reason why Limine is so much slower is because my project doesn't really have any solid filesystem drivers yet - all it does is try to find and read a couple of files which I don't think exceed 50-60KB
It becomes pretty slow if I add in a few MB lol

[wary pecan]

Limine is slower, i can accept that

but i was mainly talking about Flanterm itself

being faster

plus Limine is still 3 times faster than GRUB as per your findings so, i'll take it :p

[small field]

Fair

[small field]

Added formatting functions; this part is pretty much 100% done at this point, thank god

Example of EFI GOP boot

Example of BIOS VBE boot

[small field]

EFI and BIOS text mode boot respectively

The InfoTable is the information passed on by the bootloader stage, the kernel just initializes graphics mode / text / etc. and displays the values

[small field]

I just noticed I haven't left a bootable image file anywhere, so here's the latest one if anyone's interested
Supports both BIOS and UEFI boot, though x64 is a requirement

[small field]

I replaced the github README image with this, it's pretty good

[small field]

I revamped the usable memory map so that it's completely, 100% usable for the kernel now
Previously it was more limited, because while it did contain usable and non-overlapping entries, they weren't always guaranteed to be page-aligned, and they also contained data allocated by the bootloader (the point of InfoTable::Memory.PreserveUntilOffset was to tell the kernel "hey, don't allocate anything after this specifically")

• The page aligning part is important because I think that may have been crashing some systems as well; I'm yet to confirm that, but it would make sense

Now though, it's guaranteed to only contain memory areas that

• Are marked as free memory (Type == 1 for E820, Type == EfiConventionalMemory for EFI)
• Are guaranteed not to overlap with any other entries
• Are guaranteed not to be used by the bootloader (or the firmware) at all
• Are guaranteed to be page-aligned (to 4 KiB)
• Are guaranteed to be over a certain size (16 KiB, as defined by MmapEntryMemoryLimit in Common/Common.h)

[small field]

In other words, it's pretty much guaranteed to just work, out of the box
(Obviously a faulty bootloader can report wrong values here, but my kernel entrypoint also checks most of these claims, which has really helped avoid bugs)

In fact there's an entire group of return values just for memory map issues:

    // (High bit is 6h - issue with memory map-related data)

    EntrypointMemoryMapIsNull = (6ULL << 32),
    EntrypointMemoryMapHasNoEntries,
    EntrypointMemoryMapHasOverlappingEntries,
    EntrypointMemoryMapNotPageAligned,
    EntrypointMemoryMapEntryUnderMinSize,
    EntrypointMemoryUnderLimit,

    (const char*[]) {

      "The pointer to the bootloader-provided memory map is invalid.",
      "The bootloader-provided memory map appears to be empty.",
      "The bootloader-provided memory map has overlapping entries.",
      "The bootloader-provided memory map is not page-aligned.",
      "The bootloader-provided memory map has entries under the size limit.",
      "The system doesn't have enough memory for the kernel."

    },

[small field]

This is really really useful because I'm planning on implementing a buddy allocator and this makes it totally effortless
Up until now (and in the bootloader, which is why PreserveUntilOffset even existed) I've just been using a waterfall/watermark allocator, and hoping that the stack has enough space, which isn't great

[small field]

8,800 lines of code (14,200 including comments and 20,700 including blanks), wow!!

Not to mention close to 100 files

[small field]

Working on an allocator

(This is a weird mix of a buddy allocator and a stack allocator using linked lists, which sounds crazy, but in theory it should have O(1) best-case and O(log n) worst-case time complexity for allocating and freeing memory, while also avoiding fragmentation and being able to support different sizes)

[tribal trout]

i just looked up free95 on google, and man what the fuck

ive already found like 3 articles talking about it 😭

all of these articles feel like ai wrote them, 99% of the stuff they're talking about is probably pulled from their a**

wait this shit also has a subreddit? 😭

708 stars 💀

[small field]

Yeah its a meme, they switched kernels though

I want to try the old one with the GUI now

[tribal trout]

lmao

[tribal trout]

good luck, they dont even build it right

they did not disable the stack protector nor mmx or sse

ok this is absolute junk the gui shows for one frame and then disappears

[small field]

lol what

[small field]

Malloc() and Free() works totally fine for my simple, but it's still incomplete

• The free function takes in a size parameter and then doesn't check it, which is.. the easiest way to do it but certainly not the best
• This still only picks out blocks of a certain size - it can't break up a 8 KiB block into two 4 KiB blocks if there are no 4 KiB blocks available and vice-versa, it just attempts to find a single 4 KiB block

It's fast though - O(1) malloc and free in this setup

[small field]

It works for me but the window is constantly flashing because they don't double buffer

It is faster because of the SSE optimizations thankfully

oh my god

[tribal trout]

ah

[tribal trout]

😭

jesus christ

[small field]

From what I saw it's probably just a young teenager, if anything this is kind of impressive for that

(and I say that as a 17 year old)

[tribal trout]

oh, i wont say my age but its definitively not what most people expect

[small field]

With my current design I can't do much about the size requirement for Free(), even if it means it's unstable / vulnerable
So, just in case, I'm forcing the caller to pass in the length as a const uintptr*, as well as adding a few extra safety checks to Free() (including returning a bool that tells whether it's been freed without any problems)

Instead of

void* Area = Allocate(10000);
Free(Area, 10000);

I'm explicitly forcing

const uintptr Size = 10000;
void* Area = Allocate(&Size);
if (Free(Area, &Size) == false) {
  Panic("blablabla");
}

It's still not impossible to cause problems, but this makes it a lot harder to accidentally cause them

[small field]

I added a few extra features - namely, each node in the linked list is now linked in two ways:

• It's doubly-linked in terms of position (which is to say, each node has a previous and a next node, corresponding to the direct position in memory)
• It's singly-linked in terms of size (which is to say, each node that corresponds to a certain block size is linked with each other)

[small field]

This raises the question "how do you add a new node / free memory without traversing the entire list", and that's where the magic comes in: you don't; instead:

• Each usable memory area is divided into a reserved segment (which contains the nodes themselves) and a usable segment (which contains the memory you actually want to use)
• Each block in the usable segment has a unique position within the reserved segment
• So, in order to find the previous and next nodes, you just check if there are any nodes in the reserved segment at (position - n) and (position + n) until you find one, or leave it out as NULL otherwise

Which is to say - instead of traversing the linked list to see if there's a node, you know that, if there is a previous or next node, that it's going to be located at a few offsets from the predicted position of the node, so you just check those

[small field]

This requires the reserved area to always be clear, and for any allocated nodes to be cleared out / deleted entirely
Thankfully I already have a fast enough Memset() (and a small enough reserved area) that this doesn't appear to be an issue though

I don't know how slow this is in the long term though, since it has low time complexity (worst case O(log n), best case O(1)) but high overhead
However it is space-efficient

(It's also not well documented at all for now, it does work however)

[small field]

(It's also worth mentioning that this is not a PMM - in fact the kernel is made to run exclusively in identity-paged mode, to keep compatibility with EFI)
(However, it still has support for different sizes, so it cannot just be a single stack of 4KiB blocks for example - by the end, the goal is to be able to allocate any block size (as long as it's a power of two above the page size) in O(log n) tme

This is the same for 4KiB blocks as it is for 4TiB blocks, and once I get this working, it should also be able to do things like break up a 16 KiB node into 8K+4K+4K if there are no 8KiB or 4KiB blocks left (in O(log n) time as well!)

[small field]

9,000 lines of code!! (14,500 including comments and 21,400 including blanks)

[small field]

Ok I'm close

[small field]

This is making me realize, if I don't deallocate objects in the exact opposite order, it may not join them properly, leaving it fragmented

uhhhhhhhhhhhhhhhhhhhhhhhhhhhhhb

[small field]

It only looks in one direction for merging/coalescing blocks, meaning that if I divide a block into [a][b], and then free [b] first instead of [a], it won't attempt to merge [a][b] into [ab] again when I free [a]

What I can do is look both ways though, hmmm

[small field]

This only happens with "left" blocks, so I may not have to do that, but I'm not sure

No no I don't think I can determine that

[small field]

I can do this but it would require bounds-checking, which requires adding more information to each node, which requires using more space
The alternative is looking up the entry each time, but it's quite time intensive

[small field]

Actually no I don't, my list is doubly-linked, I just need to look forward and it's guaranteed to be NULL if it's the end or out of bounds or something

[small field]

Okay, I fixed it

[small field]

I found new bugs which I had to fix (one of which unfortunately involves making the node size a lot bigger = significantly slowing down initialization time)
But I'm close, I can allocate and free well 90% of the time at this point

(I can allocate properly 100% of the time, I just can't free properly sometimes; it keeps either removing or adding an unrelated entry when I try to merge blocks)

Im tired man

[small field]

I fixed it!!
It works perfectly now, I have a fully functioning allocator omg

[small field]

It's not the most efficient thing in the world, and has a lot more overhead than a regular buddy allocator, but

• It works just fine across multiple memory areas
• It's guaranteed to be able to allocate and free in O((log n)^2)
• It supports multiple allocation sizes (from the system page size, all the way until the 64-bit limit)
• It guarantees that all allocations are physically contiguous
• It automatically rearranges itself to always have the most efficient configuration / avoid fragmentation
• It doesn't use any external memory, including stack memory

(The kernel is guaranteed to be identity mapped, so it can't really use virtual memory tricks either)

I'm in awe, I've been debugging this thing for so long that it's strange to see it actually work

https://github.com/NunoLealF/Serra/blob/main/Common/Kernel/Memory/Allocator.c

[storm fable]

what protocol is serra planned to support?

[small field]

My idea is to just start off with loading EFI applications (which isn't that difficult), and then hopefully implement multiboot/limine/etc.

As far as I can tell it already has enough data to implement the protocols themselves, it just doesn't have anything other than a (very simple, bootloader-only) FAT driver so it can't really load anything for now

[small field]

I did some performance tests, and on Bochs:

• It takes around 400 cycles to allocate and free a block that's immediately available
• It takes around 800 cycles to allocate and free a block that's not available ||(meaning that it has to be carved up from a larger block when allocating, and then merged when freeing)||

In general, it seems like it can do 20k-50k allocations and frees per second at the very least, and 2-5 million on my modern i5 laptop

This isn't that fast ||(from what I can tell it's half the performance of Linux)||, but I'm still happy with it

[small field]

That being said, the performance on QEMU KVM - with immediately available blocks - is nearly identical to Linux malloc()

• It takes 1.75 seconds to run 100,000,000 iterations of free(malloc(Size)) on my native Linux system
• It takes 2.25 seconds to run 100,000,000 iterations of Free(Allocate(&Size), &Size) on Serra's allocator on QEMU (with KVM)
• It takes 9.5 seconds to run 100,000,000 iterations of Free(Allocate(&Size), &Size) on Serra's allocator on QEMU (without KVM)

[small field]

I did some tests on real hardware and I'm frankly just baffled

[small field]

Older systems - especially BIOS systems (it tends to be a lot slower on BIOS) are really slow
I think it probably has to do with how I implement Memset(), but if anything only enhanced rep movsb/stosb systems seem to be getting a speedup, lack of ERMS support is what's making it slow
I only memset small values inside the allocator, so it always uses rep stosb anyways, ERMS or not

[small field]

It took me less than a second to run 10,000,000 iterations of Allocate() + Free() on my i5-1240P UEFI laptop, but almost 7 minutes to do the same thing on my i7 860 BIOS system

On another system (with a 4th gen i7), it ran in less than a second on UEFI, but took a lot longer on BIOS

[small field]

Also, just for reference - this isn't just graphical, it also has fallbacks for EFI and VGA text modes

You can toggle them by either setting Graphical := false in makefile.config, or booting on a system that doesn't support graphics mode at all

[small field]

After some testing I've successfully implemented double buffering, and optimized the print and memory routines a fair bit

[small field]

My memory allocator still isn't faster, but I've optimized my printing functions a fair bit, and scrolling is universally faster (still not that fast on most systems, but still reasonably fast imo)

(I also removed fxstore/fxrstor since it obviously made SSE/AVX loads slower, and reduced the threshold for SIMD memcpy/memset to 64 bytes instead of 2048)

[small field]

After a few hours of work, I can now read from the disk on BIOS systems with int 13h

Here's a recording I took of a demo that displays the first 1 MiB of the boot disk (qemu-system-x86_64 -cpu qemu64 -m 256 -drive file=Serra.img,format=raw -enable-kvm)
-# Disclaimer: Your speed may absolutely vary, scrolling is much much slower than this on real hardware

[small field]

In VGA text mode:

(the speedup in the middle is me switching out of power saver mode on my laptop)

[small field]

Once I add an interface for EFI_DISK_IO_PROTOCOL/ EFI_BLOCK_IO_PROTOCOL, I can then add something similar for EFI, and build a common disk subsystem on top of that
It's not perfect - especially given the lack of choice - but it should give me a good foundation for my disk drivers

If I ever implement proper IDE/AHCI/NVMe/eMMC/USB/etc. drivers, I can just add them in as necessary

[small field]

I'll test this on real hardware, and then I'll be back

[small field]

It works universally as far as I can tell
In the meantime I've been working on an EFI disk driver as well (using EFI_BLOCK_IO_PROTOCOL, or efiBlockIoProtocol in my code)

[small field]

I can now read using the EFI block IO protocol just as I can with int 13h
This is the exact same data being read with ReadDisk_Bios(Buffer, 0, 2048, (DL drive number)) on a SeaBIOS system, and ReadDisk_Efi(Buffer, 0, 2048, 0) on an OVMF system

[small field]

Better, here's the same output with the info table visible (see how the first has InfoTable::Firmware::Bios and the second has InfoTable::Firmware:Efi)

[small field]

After some testing, this seems to work on real hardware, with the caveat that a lot of the efiBlockIoProtocol devices don't work
I mean, a good chunk of them do, including the important ones, but they're not actually ordered in any reasonable way

[small field]

I added some logic to just automatically pick the first protocol that works, and that's worked on all of the machines I've tested, with the caveat that 90% of the time it's not actually reading from the boot drive
(Even when I make sure the boot drive is always first, it still never works, usually EFI_MEDIA_CHANGED)

I don't mind though, since that just means I can't find the boot drive easily, and that the demo sometimes won't actually read from the correct drive (but it'll still read)

[small field]

(NEVERMIND, ignore all of this, I was making a mistake)

[small field]

Okay after fixing the issue I can say that it works perfectly on most hardware
(The one exception being a PC I have where it seems to read totally garbled data for some reason, but that's just one disk)

(I could be wrong on that too, I'm not sure if it's just reading from the start of a filesystem or something)
Macs work just fine though!

[small field]

After a few hours of work, I now have volumes (and a standardized format for accessing them, method()drive()partition())

  {

    // [The method needed to access this volume, as well as the drive
    // and partition number - `method()drive()partition()`]

    enum : uint16 {

      // ('Universal' volume types that correspond to `DiskInfo.BootMethod`,
      // and that can be read from at boot-time)

      VolumeMethod_Unknown = 0, // (You can probably ignore this)
      VolumeMethod_EfiBlockIo, // (Can be accessed via efiBlockIoProtocol)
      VolumeMethod_Int13 // (Can be accessed via the int 13h wrapper)

      // (Additional volume types that can be set up later on for
      // specific types of devices - TODO)

    } Method;

    uint32 Drive; // (The drive number (implementation varies))
    uint16 Partition; // (The partition number, or 0 if raw/unpartitioned)

    // [Information about the volume's type (and partition, if one exists)]

    enum : uint16 {

      // (If this volume represents an entire device, show the partition
      // map type (for example, MBR or GPT))
      // `Partition == 0`, bit 8 is cleared.

      VolumeType_Unknown = 0, // (Couldn't determine type)
      VolumeType_Mbr, // (Appears to be a 'raw' MBR volume)
      VolumeType_Gpt, // (Appears to be a 'raw' GPT volume)

      // (Otherwise, if it represents a partition, show the type)
      // `Partition != 0`, bit 8 is set.

      VolumeType_Fat12 = (1ULL << 8), // (Appears to be a FAT12 partition)
      VolumeType_Fat16, // (Appears to be a FAT16 partition)
      VolumeType_Fat32, // (Appears to be a FAT32 partition)

    } Type;

    uint32 MediaId; // (`VolumeMethod_EfiBlockIo` only - the media ID.)
    uint64 Offset; // (The LBA offset of the partition, if applicable)

    // [Information about how to interact with the volume]

    uint16 Alignment; // (The alignment requirement for a transfer buffer, *as a power of 2*)
    uint32 BytesPerSector; // (How many bytes per sector/block?)
    uint64 NumSectors; // (The total number of sectors/blocks in the volume)

  };

[small field]

This is totally overcommented, but the point is that it's flexible; I can add many different disk types to this and it should work just fine
If I ever want to write an ATA PIO or NVMe or USB driver, or if I ever want to add support for ext4 instead of just FAT, this makes it really easy to work with

(I can also have volumes represent specific partitions instead of disks, since that's a requirement of EFI_BLOCK_IO_PROTOCOL)

[small field]

The only thing I have to do now, is to add a 'common' read from disk function ||(up until now I've just been relying on individual functions for each type)||, and then implement a simple filesystem/MBR/GPT driver (to also fill out the volume type)

After that, I should have a modular disk driver that I can use on both BIOS and UEFI, and that's easy to expand in the future

Additionally, on EFI, this also reads all disks (because of how efiBlockIoProtocol works), so I'm not even limited to the current boot drive - if there's an EFI System Partition anywhere on the system, I should be able to detect it (meaning I can put it on a USB, and have it go through the main system drive's filesystem, for example)

[small field]

This.. took me a lot longer than it should have

But it's here, and it works really well

• It can automatically deal with buffer alignment restrictions
• If a buffer is needed, it minimizes memory by allocating in-place unless absolutely necessary
• Like EFI_DISK_IO_PROTOCOL, it works with individual bytes which don't have to be aligned to anything
• It works on both BIOS and EFI (obviously)

(I also added Memmove(), since it was necessary)

[small field]

It feels pretty fast too, though that's to be expected
(Even int 13h doesn't feel slow, which is a pleasant surprise!)

Here's how fast it boots on QEMU KVM + OVMF
Each one of these walls of numbers comes after a disk has been loaded (which is to say, there's one of these every time a disk is loaded)
As far as I'm aware even slow hardware can do ~1 MiB/s, KVM can probably do hundreds of times that - the main bottleneck is displaying the data

[storm fable]

is this using a custom EFI library?

[small field]

Yeah https://github.com/NunoLealF/Serra/blob/main/Boot/Efi/Efi/Efi.h

This isn't a complete EFI or UEFI implementation by any means ||(in fact it's the absolute bare minimum)||, but it was still a lot more simple than trying to integrate GNU-EFI or EDK-II
And once you get the hang of it's honestly not that hard

[small field]

I don't use a call wrapper like GNU-EFI does, since:

• My EFI loader is compiled with the Microsoft ABI (on x86_64-w64-mingw32-gcc), so it's natively available
• My kernel is compiled with the cdecl ABI (on x86_64-elf-gcc), but the compiler supports __attribute(ms_abi)__, and it works

(I do modify CR0 and CR4 to enable SSE2/FPU features, but the UEFI Specification is chill with that)

[wary pecan]

Nyu-EFI is pretty seamless and it allows you to use normal ELF compilers

[small field]

Oh that looks interesting

It doesn't make any sense to switch now, since my current system is working
But honestly if I had known about it I would've used it earlier, just looking through the repo it's a lot cleaner

[small field]

On another note, before I go to bed, I did two things

• I tested this extensively on real hardware, and with the exception of one machine that just seems to have a completely broken implementation, it works well
• One thing I did notice was that it wasn't booting on my main PC though, and after some investigation, I realized I wasn't dealing with the LCMV bit in IA32_ENABLE_MSR properly - that's now been patched
• • Long story short, but it seems like recent Ryzen CPUs don't actually have that MSR, so it just crashes whenever you try to access it - I assumed it was architectural, and it even seems to work fine on older AMD CPUs, just not newer ones, so I had to work around that by checking for MSR support first, and then looking at the max CPUID leaf
• • However, it seems like those same CPUs also don't need the LCMV bit to be cleared in order to access higher CPUID leaves (past 3h), but just in case I did add a check for that

Now it boots on Ryzen which makes me happy

[small field]

You can see MSR 0x1A0 (1A0h, 0000_01A0h) isn't supported, and that (CPUID leaf 0h).EAX == 10h (which is more than the 3h limit that LCMV bit disabled CPUs usually have)

[lime hemlock]

why do you even need to touch LCMV?

[small field]

To check for xgetbv/xsetbv support, which requires leaf 0Dh
(I just use it to set up AVX, but it's totally optional - the only reason I do is because I have optimized memcpy/memset functions for SSE2 vs AVX vs AVX-512)

[lime hemlock]

linux gates clearing LCMV to family >= 6, and for family 6 to model >= 0xd

hm they changed the logic betwene 6.9.9 and 6.15.1

now it's just ```c
if (c->x86_vfm < INTEL_PENTIUM_M_DOTHAN)
return;

interesting, i don't think i've ever seen this before

[small field]

Huh that's interesting

[wary pecan]

hm

i'm pretty sure i never needed to touch LCMV in my life and AVX worked fine on Ryzen systems

unless this is about using AVX in protected mode, in which case i have no idea

[small field]

It's only an issue on older systems, I think - the bug was from touching LCMV at all when Ryzen didn't support it (or need to, really)

According to sandpile.org (and a few other websites - I think there's one that talks about reverse-engineering the Windows kernel specifically), the maximum standard CPUID leaf is apparently limited to 3h on some older CPUs in order to keep compatibility with Windows NT (which didn't trust anything over 3h)

Which is kind of important because I use xsetbv/xgetbv in order to deal with XCR0 on AVX systems, and even though it seems like that's enabled by default basically everywhere, in order to check support for it I believe you have to check CPUID leaf 0Dh

[small field]

(I only even figured this out because it was crashing if I ran QEMU with -cpu Opteron_G3, which has AVX but not XGETBV for some reason)

[small field]

On another note, I'm starting to work on a partition/filesystem driver now
For now it seems like it detects MBRs just fine - I don't think there's any explicit logic that lets you figure out if a volume contains an MBR or not ||(in this case, because of EFI_BLOCK_IO_PROTOCOL's behavior, a volume can be a single filesystem/partition or an entire disk)||, so I just run the following checks:

• Check if I can read from the volume at all
• If I can, see if the bootsector signature (AA55h) is present
• If it is, go through each partition entry, and calculate the start and limit in bytes
• • I use LBA values (with the actual sector size) if possible, but otherwise, CHS addressing with 63 sectors per head, 16 heads per cylinder, and 512-byte sectors
• • I also skip this step entirely if the type is 00h
• Go through each partition entry's limits and run the following checks:
• • If all four entries are zero/empty, I assume an MBR isn't there
• • If an entry isn't zero, but starts or ends at 0h, I assume there isn't an actual MBR
• • If an entry's limit exceeds the actual drive limit, I assume there isn't an actual MBR
• • If an entry's limit is before its start, I assume there isn't an actual MBR

[small field]

This works just fine for detecting my boot drive's partitions (both MBR and GPT)
-# Though this example is GPT (you can see from the EEh partition type)

But I honestly don't know if it's safe - there's probably some combination of code out there that could get mistakenly detected as an MBR
(I can add an extra check for the LogicalPartition flag for EFI_BLOCK_IO_PROTOCOL though; that way I'd be able to skip anything the firmware says is a partition and not a whole disk)

[small field]

I added that, hopefully it's safer now

[storm fable]

interesting

[storm fable]

Why not using clang?

[small field]

I

Probably should

(The code itself is still clang-compatible as far as I'm aware, but the build system uses gcc)

[storm fable]

ah

clang has really nice cross compliation

this deserves more stars tbh :^)

[small field]

It really does!! It would be perfect for this, I don't know why I forgot about it until now :(

(I've heard rustc is similar, but I'm not sure - maybe if I ever learn Rust I'll make the switch to something similar)

[small field]

Thanks <3

This should actually be done relatively soon
I have to turn it in by Friday, please send help

[small field]

(I've been working on this for a while now, but at the beginning of the school year I thought I would make it my senior project.. and the deadline for that is June 13th)
(Way more stressful than I was anticipating haha, but it's been a fun journey)

[storm fable]

and you are doing this alone!?!?!?! even more impresive

[small field]

10,000 lines of code!!

Aaaaa

[small field]

16,000 including comments actually

[small field]

I can't believe I only noticed now, but I've been debugging this issue for a few hours now, and I only now figured out that it's because I didn't ever set up relocations

(And it still works fine on -O0, even without those)

[small field]

I've literally been reading variables from the wrong addresses this entire time and it only popped up after like 4,000 lines of code

Frankly I'm kind of amazed

[small field]

Now it works!!!

Once I get around to adding - and interpreting - partitions to the volume list, I should be able to read files
At which point I can actually make this a proper boot manager

(I can already read MBR partitions just fine, it's just GPT partitions that need to be taken care of)

[small field]

Btw, just to clarify:

• Devices booting using BIOS or CSM can only read the boot drive, for now
• Devices booting using EFI can read any drive on the system that has an associated EFI_BLOCK_IO_PROTOCOL (= really any block device on the system)

[small field]

In this screenshot, volume 0 represents the boot drive (which is just Serra.img), but volumes 1 through 3 represent actual drives on my system that I added with QEMU
-# Volume 1 is just a USB drive with the same image as volume 0, which is why it's identical
So it definitely works

Here's what a USB drive with three MBR partitions looks like, on UEFI (left) and BIOS (right) boot

(layout)

[small field]

I'm going to try to finish DetectPartitionMap() tomorrow (so it locates and opens a volume for each partition), then that should be it
I have to turn this in really soon, so.. even though it's not a boot manager at this point, I don't have much of a choice
||(In order to turn this into an actual boot manager, I'd need to implement a proper filesystem (at least FAT) driver, implement the necessary boot protocols (native firmware (EFI app / VBR boot), multiboot/Limine/BOOTBOOT/etc.), implement keyboard/mouse drivers, and show a menu of some sort)||

• ||(That's actually not too hard, all things considered - the kernel is structured in such a way that it would be relatively easy to implement all of this - I just don't have time)||

I don't know if I'll have enough time to finish this after I turn it in, since I'll be busy studying for national exams (and potentially moving to another city for university)
But all things considered I really liked working on this, I learned a lot and I hope I'm able to get a good grade despite it being unfinished

[small field]

Tomorrow afternoon (around 24 hours from now), I should publish a build on GitHub, with updated instructions and such

For now, I'll just try to get a good night's sleep, so I have the energy to put in the work tomorrow

[wary pecan]

❤️

good luck with your future endeavours

[small field]

Thank you ❤️

[small field]

I can now read MBR and GPT partitions
(I am yet to actually add them to the volume list, though - I also haven't tested backup partitions yet)

[small field]

I've tried to, but OVMF fixes them automatically lol

[small field]

Done, took me a lot longer than I was expecting haha

(type 102h == FAT, type 101h == basic data partition (Linux or Microsoft), type 100h = unknown/unsupported)

[small field]

It's been a while, but I added GOP BLT support (for scrolling)
Ironically this is actually a fair bit slower on some PCs, but on others it's much faster

I also optimized the printing code to minimize the amount of framebuffer operations

[small field]

It's not incredibly fast or anything, but even on my slowest testbench it can scroll once every 0.5 seconds, whereas before it used to take close to 10x that

(Though some other testbenches went from scrolling 4-5 times per second to 1-2, so it's noticeably more laggy there.. but I'd rather have that over most machines being fine and some being unbearably slow)

[small field]

Oh my god

[tribal trout]

mfw

[wary pecan]

btw you can use -M q35 so ovmf doesn't take 2 and a half hours to boot

[small field]

Oh thanks

[wary pecan]

np

[small field]

Added partial Clang compatibility
It's still a work in progress, and not really ideal because it doesn't have full C23 support / has a couple of weird behaviors
But it does work! (Literally the only addition I had to make was add -mcmodel=large to the compiling options, otherwise it works perfectly (despite the 500 warnings))

/Boot/Efi still needs MinGW but it seems like lld has trouble with PE executables anyways, at least you can literally just sudo apt install binutils-mingw-w64 gcc-mingw-w64 instead of cross-compiling it

[storm fable]

I have never had issues with clang and lld when doing UEFI stuff, so your flags are prob wrong or I am wrong :^)

[wary pecan]

Nyu-EFI :^)

[small field]

It just doesn't seem to like -Wl,--subsystem,10 in the link options
I can probably fix it, I'm just not familiar enough with clang to (and also too lazy)

[storm fable]

https://git.piraterna.org/Piraterna/aurix/src/branch/main/boot/platform/uefi/Makefile

[small field]

The target was the issue - it also really doesn't like typedef'd unions for some reason..? But I managed to fix that
(I was doing x86_64-w64-mingw instead of x86_64-unknown-windows)

However

It's complaining about a -nologo option that's automatically injected by clang on linking, but that doesn't actually exist..??

(Not your project, I mean mine)

If I add -fuse-ld=lld-link it goes slightly further, but it still fails to link because it can't figure out what an .o file is (!?)

[small field]

It also discards like half of the parameters I throw at it (including -ffreestanding and -O3) sometimes

(This is only when linking with clang, not lld-link, which could work, but my build system isn't made for that)

[small field]

Using $< instead of $^ fixes it, but then it can't resolve any symbols (even though they're obviously there)
I don't think it can actually do freestanding PE applications reliably; -nologo for example is Windows-specific. I'd need to rewrite the build script to use lld-link I think

[small field]

I just realized I never tried to do fizzbuzz

[small field]

I was wondering why my framebuffer was so slow on BIOS
Turns out I was accidentally mapping it with the cache disable (PCD) bit, now that I fixed it the performance speedup is absolutely massive
I honestly can't believe I actually did that

[tribal trout]

that reminds me of a friend that accidentally mapped his kernel as NX, thinking that NX meant Execute Enable

[small field]

Scrolling is still slow on something like Bochs, but printing is even faster than the VGA console

And on QEMU KVM both are so fast that I can scroll and print several hundred lines per second

[small field]

Actually it's so fast that if you do something like this

  for (int a = 0; a < 100000; a++) {
    Print("1 \n\r", true, 0x0F);
    Print("2 \n\r", true, 0x0F);
    Print("3 \n\r", true, 0x0F);
    Print("4 \n\r", true, 0x0F);
    Print("5 \n\r", true, 0x0F);
    Print("6 \n\r", true, 0x0F);
    Print("7 \n\r", true, 0x0F);
    Print("8 \n\r", true, 0x0F);
    Print("9 \n\r", true, 0x0F);
    Print("0 \n\r", true, 0x0F);
  }

It syncs up to the refresh rate / shows several numbers at once, like a superposition

[small field]

So after a little bit of testing

(And bugfixing, i had a bug with my bios loader)

[small field]

This doesn't really apply to EFI since the firmware overhead there is too large
But I can boot Serra graphically off of just 3.5 MiB of RAM

(See the 'found a 1000h-sized block' message? That's the system saying it literally only has a single free page left)

[small field]

That goes down to 2.5 MiB if I turn off graphics mode

[small field]

For full resolution, it needs about 8 MiB or so though

[small field]

In reality this basically doesn't mean anything because 64-bit is a requirement and I seriously doubt there's a single x86-64 machine in existence with that little RAM

That being said
I do plan on revamping the memory management system in the future, and actually porting it to 32-bit
The reason why I went for 64-bit in the first place is totally moot now because of other design decisions

[small field]

In theory the BIOS loader already runs on anything from Pentium Pro (i686) onwards, so that could be the minimum requirement

No matter what though; over the next few weeks, I'll try to rewrite as much as I can, and fix any bugs I've encountered up until now (int 13h still crashes on some systems for example)

After that I can start working on other core things (actual exception support, better HAL, dual 32- and 64-bit support, etc.) and then get to reading files

[small field]

After thinking about it
I think I'm going to do a full rewrite (though I will still reuse as much code as possible)

I did many things right this time around, but there were still quite a few design decisions that I could have honestly done better

I'll still reuse most of my existing code, but I'll try to go about it a different way - I want this to progress beyond just a hobby boot manager and into something people will actually use (e.g. Limine or GRUB), and for that, I'll need to refactor a lot of things from scratch

[small field]

For now, the focus will be on putting out a 'final' release with some last few bugs polished, and then move that to an 'old' codebase

I'll still keep the same goals (and many, though not all of the same design decisions) in mind, and it will keep the same name and logo
But my aim is to:
1 - Provide better documentation (something like a user guide that explains the boot process in detail)
2 - Come up with a better high-level design across pretty much every part of the boot manager
3 - Implement support for multiple architectures (both 32- and 64-bit, possibly even ARM or RISC-V in the future)
4 - Reimplement existing features from scratch in a more extensible/reliable way
5 - Revamp the build system to be more acommodating of different compilers (especially clang/lld) and environments (maybe move away from makefiles? I believe skiftOS did that)

I also genuinely want this to be easy to understand, and to serve as a reference in the future
That means high quality code, but also really high quality documentation and exception handling, so:

• From now on, the documentation will explicitly line everything out, like a specification - what BIOS calls are used? What EFI services are used? When, why, and how?
• Not only that, but exceptions will be explained in detail; if a user encounters an error, I want them to know exactly what happened, exactly what might be causing it, even if it means adding in a lot of extra detail
• • In reality, this probably means doing it the Microsoft way and having error codes that you can then look up in a manual (either that, or you totally discard multilanguage support, which I also want to add)

It probably also means that my progress will be slow and cumbersome so I just hope it pays off

[languid barn]

best of luck, thats a big move

[small field]

Thank you <3

[small field]

Rewrite hasn't started yet, but I've fixed some bugs; at this point I think it's ready for the "final release" before I start over again

[small field]

More specifically:

• I fixed several linking bugs, and added lld support (actually tougher than I was expecting, I think I might switch to PE since ELF has so many weird things you have to account for)
• • Everything apart from the EFI bootloader can now be compiled on both GCC/ld and Clang/lld without any issues
• I also fixed a bug where I wasn't properly preserving registers when switching in and out of long-mode for the fallback int-13h driver
• • Also one that would cause issues with some low amounts of RAM (due to a rounding error) where it wouldn't properly identity-map everything

As well as a display driver issue

[small field]

Right now it works on every system I have and quite reliably at least

What worries me is that the build process itself is fragile:

• Cross-compilers are hard to set up, and although clang really simplifies the process, it also isn't well documented at all
• It is really hard to set up something that works with both GCC/ld and Clang/lld, to the point where I'm considering either switching to PE, or forcing the use of GNU ld
• Makefiles are really bad and don't scale well at all for anything involving more than just compilation, but I'll probably switch over to Meson for the rewrite

[small field]

Like, maybe it's a skill issue on my part, but lld is horrible to work with

[small field]

After thinking about it for a while

[small field]

I've been looking into Secure Boot, code safety, etc. - as well as researching how to do cross-compilation better, and supporting different compilers - and after thinking about it for a while, there are two more changes I want to add to the rewrite, that I think would make it significantly safer and more extensible

(1) First of all, in order to implement some safety features (such as ring 3 on BIOS systems), and to be able to better support different architectures, I intend on making the core part of the boot manager work like a microkernel
Which means that, instead of having functions directly interact with firmware services, they instead communicate through a platform-independent layer that handles things for them; this applies not only to BIOS or UEFI-specific quirks, but essentially everything that can be abstracted away
Not only should that make it easier to port to different architectures, but it also helps to isolate individual parts of the code (which is crucial, as it really helps with verifying the kernel API always has the same behavior no matter what)

This is also important as it means the rest of the code can't directly call kernel symbols, which means that functions can be limited only to the functions which they need to call (at the syscall layer)
That doesn't protect against attackers, but it does help protect against bugs and memory safety issues, at the cost of performance (which is admittedly an issue.. but this is an optional feature)

[small field]

Which brings me to my next point

(2) This is still a recent decision, so I could go back on it any time, but I do plan on switching to Rust

To be clear, I'm not a furry or anything, and this decision isn't necessarily motivated by memory safety - you can implement many of the same features in C at runtime and still achieve far better security than something like GRUB
But over time, I kept finding out that Rust just.. handled a lot of these problems in a much better way. Instead of checking for issues at runtime, or implementing your own custom borrow checker (I actually didn't know what it was, I came up with the idea and then found out Rust had that built in!), Rust handles a lot of that for you.

Obviously there are concerns with compatibility, and especially code readability (Rust's code style is genuinely really ugly)
But what it lacks in readability, it makes up for in implementing many really useful features for this sort of secure low-level code, and in being much better designed than C overall
At the end of the day, my current codebase is C23 and relies on GCC, which means that you have to (a) cross-compile several versions of gcc, and (b) implement only a limited set of C23 (as GCC doesn't have full support for it), while limiting yourself to extremely new compiler versions, and missing out on a lot of comfort features that are in C++ and Rust

Rustc can cross-compile natively, and handles a lot of things that are otherwise not well specified in C (explicitly sized integers (u32 instead of unsigned something??), compiler versions, etc.), while implementing a much more modern/C++-like featureset, which is genuinely crucial for the kind of code I want to write
(And all of this without C++'s half-baked and annoying-to-implement requirements for freestanding code)

[small field]

From my perspective, the alternative would have been:

• For C, either support only a handful of new compiler versions for C23 support, or remove useful C23 features and deal with tons of compiler-specific cruft
• For C++, all of the above x10 because it's not built for this sort of thing
• For Rust.. support a single compiler, with a single build system, while keeping most of the useful features (traits/implementations/classes/namespaces)
  -# Though I could be wrong on this.

And that's not to mention the massive safety boost, which is genuinely crucial for ring 0 code; I don't think it's impossible to write truly safe (or at least, safe enough code) in C, but the fact that Rust forces it on you on compile-time does really help, especially if I ever want to implement Secure Boot support in the future (or even attempt to get my stuff signed by shim/MS)

With that perspective in mind, I don't see any other way forward but Rust

[small field]

I was also planning to implement ring 3 support on UEFI systems, as it is feasible on spec-compliant systems, but as far as I'm aware it hasn't been done before; it would depend on

• Whether code can ever be preemptively interrupted by anything but an actual interrupt (like SMM), and whether it could change the page tables in some way
• Whether all UEFI implementations actually identity-map memory (there isn't anything stopping them from not doing so, as long as they copy everything back after ExitBootServices())
• Whether all UEFI implementations actually follow the specification when it comes to (1) not preemptively interrupting code outside of timer interrupts or firmware calls, and (2) even executing code sequentially in the first place

And in practice, you'd need to be able to handle some types of interrupts on non-x86 platforms ||(x86 has syscall/sysexit/etc. which doesn't require an IDT)|| for things like syscalls (how else are you going to switch to kernel mode?), which is an issue because the UEFI specification doesn't guarantee anything about how interrupts are routed
There's no guarantee that, say, the timer interrupt won't be routed to the double fault vector, or that the system doesn't lazily map memory on page faults, or that the memory containing the IDT is even mapped (which it might not be, for platform protection purposes)
There are ways around this, they just mean you can't set up your own interrupt handlers, or that you need to rely on arch-specific hacks which isn't ideal

(None of this is an issue on BIOS, or after exiting BootServices(), but good luck loading an EFI application in that environment lol.)

As far as I'm aware none of the existing boot managers try to do this, and nobody on EDK-II's mailing list has mentioned it (other than a few people mentioning section 2.3.4.3 / "Enabling Paging or Alternate Translations in an Application", which is what allows this to theoretically be possible)

[cold hamlet]

C++'s half-baked and annoying-to-implement requirements for freestanding code
what do you mean by this? c++ mostly just works and only needs a little bit of code for specific optional runtime features
not well specified in C (explicitly sized integers (u32 instead of unsigned something??)
you can #include <stdint.h> to get the sized integer types like uint32_t (and typedef them to be like u32 if you want)

[wary pecan]

@small field why don't you help out with Limine instead? you're quite competent and i could use the help

[small field]

c++ mostly just works and only needs a little bit of code for specific optional runtime features
You're not wrong, it's just not that easy to set up, and from what I can tell it seems to be compiler-dependent (or at least poorly documented), the osdev wiki's page on it didn't explain much
But I'm glad it mostly just works, maybe I had too negative of an opinion on it; I assumed it was the kind of thing that broke really easily from what i saw.

you can #include <stdint.h> to get the sized integer types like uint32_t (and typedef them to be like u32 if you want)
Yeah i already have that, i was more impressed by how rust lets you natively specify the size of certain types and such in the config file; C/C++ still kinda fundamentally relies on the compiler doing it for you

[wary pecan]

you can sort out some of the issues for example

[small field]

Sure, that sounds good, though i'm unfortunately quite busy with university nowadays
(Though i will say i dont have experience with Rust yet, if that's what you were going for - my idea was to first rewrite some parts of my old code in rust to see how it works (basically just practice!), and then get started on a full on rewrite)

[wary pecan]

oh no

i did not want Rust related anything lol

[small field]

yeah thats fair, up until now I've only ever dealt with C for osdev lol

[wary pecan]

no hate on Rust of course

[small field]

I dont like it that much either, in this case I just think it's the most useful tool for the job
But honestly C works most of the time, I'm only even considering this because I want to plan really far ahead and hopefully avoid issues

I already have to write C really carefully because of how broken firmware can be sometimes

[wary pecan]

i am looking into building Limine with Fil-C

if you want safety of Rust but still use C or C++, it seems like a miracle cure

[small field]

That does seem really interesting (although I don't know if it works super well for freestanding code - it might, but usually tools like this tend to assume a regular implementation)

(like it talks about libc in the description, I don't know if it's a dependency or just optional)

But it does merit looking into it, tbh

Oh it's just a compiler (or compiler extension), that's pretty good

[cold hamlet]

its actually very easy to set it up and afaik most of it is not compiler-dependent

[small field]

It definitely doesn't seem that easy, but I don't have a super deep understanding of C++, so I'm inclined to trust you more on that
(Still though, what drew me to C++ was being able to deal with interfaces and classes and such, which I don't think Rust requires any set-up code for)

C shouldn't either but the standards committee would rather die than implement ::

Actually, they DID implement it! In C23, when they copied C++-style attributes, for things like [gnu::packed]
So attributes have :: but not.. The Rest Of The Language

[cold hamlet]

c++ does not need any set-up code for that either (except for pure virtual functions which needs you to add a single line: extern "C" void __cxa_pure_virtual() { panic("__cxa_pure_virtual called"); }

[small field]

Yeah i mean for the rest of the language, there's nothing needing you to set that up specifically, but in order to get that you need to use C++, which requires other setup code (even if it is quite simple)
(and i also had the misconception __cxa_pure_virtual() was a gcc/clang hack but i'm pretty sure i'm wrong on that, that's why i avoided it)

Still though i genuinely appreciate the input, I had a much more bleak idea of freestanding C++ than reality lol

[cold hamlet]

the rest of the language does not need anything at all either except for a few specific features:

• if you use global constructors you need to add a few lines somewhere to call them which is a little bit annoying but you can simply not use them
• if you use local static variables inside of functions you need to add like 20 lines of code
• if you use new and delete you need to define them (which is just 6 lines of wrappers around malloc and free)

i think thats all

if you dont use any of those you need literally 0 setup code

[small field]

Oh that's really useful to know actually, the osdev wiki talks about a lot more (setting up your own crti/crtn/crt0, etc.)
-# (Which is probably part of that? But I'm not sure)

[small field]

Also, on a funny note
I've recently been reading about Windows Internals (and about the NT kernel specifically), and it's kind of surprising how well modeled NT's architecture is for this
Like, I literally spent a few days trying to come up with a more formal definition for this (what parts will be part of the microkernel? how will hardware access be handled? etc.), only to later look up how NT works and realize that it's basically almost the exact same thing
Even the code style (UsingPascalCaseForLiterallyEverything / not even bothering to rely on the C std) is similar, I personally prefer that

It's honestly really well designed and I think the usecases it was designed for absolutely apply here (it was designed in the 90s to work on multiple architectures, and to be essentially everything-agnostic (architecture, program type/subsystem, etc.), which is exactly the qualities I'm looking for here too)
It's like the one good thing Microsoft has ever done

I'll have to turn off that one Rust flag that prevents you from using anything but snake_case lmao

[small field]

One thing is being able to run on a specific set of hardware, but being flexible enough to accommodate running on an entirely different platform that it was never designed for and had no backwards compatibility with is beyond insane
And it's the kind of thing I want to aim for too, though it's obviously a lot harder - that's also why being architecture-independent / not relying on x86 specifically, is so important

[cold hamlet]

the crti/crtn/crt0 stuff is for global constructors, except theres a much simpler way to do it. you can add the following to your linker script below where you define .rodata

    .init_array : {
        PROVIDE_HIDDEN(__init_array_start = .);
        KEEP(*(SORT_BY_INIT_PRIORITY(.init_array.*) SORT_BY_INIT_PRIORITY(.ctors.*)))
        KEEP(*(.init_array .ctors))
        PROVIDE_HIDDEN(__init_array_end = .);
    } :rodata

then, in your init function you call the constructors like this:

extern "C" void (*__init_array_start[])();
extern "C" void (*__init_array_end[])();

for (auto ctor = __init_array_start; ctor < __init_array_end; ctor++)
    (*ctor)();

(warning: this is probably for gcc/clang only, and im not sure if it differs between architectures)

[small field]

Ohhh, I see

That makes more sense tbh

[small field]

Oh my fucking god you're telling me I even do the comments the same way!?

(Here's how I tend to write comments for reference)

/* uint64 AllocateFromMmap()

   Inputs: uint64 Start - The address you want to start searching from;

           uint32 Size - The size of the memory block you want to allocate;

           bool Clear - Whether the memory block should be zeroed out;

           usableMmapEntry* UsableMmap - An array of memory map entries that
           only correspond to usable memory (the "usable memory map");

           uint8 NumUsableMmapEntries - The number of entries in UsableMmap.

   Outputs: uint64 - If successful, an address that corresponds to the end
            of the newly allocated (page-aligned) memory block; otherwise,
            zero.

   The purpose of this function is to allocate a given amount of memory from
   the system's memory map, using a simple bump allocator design, and to
   clear it if necessary.

   Essentially, this function looks for a free memory area in UsableMmap (that
   comes after Start), and either returns an address corresponding to the end
   of our newly allocated memory area, or 0 if it can't find a large enough
   area to be allocated.

   Since it doesn't have a way to free (or unallocate) memory, and isn't the
   most efficient system, this function isn't really comparable to an actual
   memory manager; however, it's *perfectly suitable* for allocating small
   blocks of memory that don't need to be freed, such as page tables.

   Keep in mind that this function doesn't have any notion of what's been
   allocated so far (and what hasn't), so if you're allocating multiple
   things, always make sure to update the starting address with the return
   value from the previous call.

   Also, keep in mind that this function *always allocates on (4 KiB) page
   boundaries*; this is pretty useful when you're setting up paging, but it
   does mean that this function can end up wasting some space.

*/

[small field]

Okay I get what you mean, I tried to rewrite a few functions in Rust in order to get a better idea of how it worked and it really isn't suitable for this
I'm considering Zig but i'll probably just switch to C23/clang and some sort of strong static analyzer with runtime checks

Like

• Strings are part of the standard library (!?)
• There's no stable ABI, other than the C one, which.. also depends on your target, it's not even really stable
• You can't have different panic functions
• You can't use anything but snake_case for some reason, at least not without explicitly disallowing it
• • On that note it also warns for a bunch of coding style violations, some of which explicitly make it LESS readable
• #![no_mangle] is being deprecated (???)
• You can't really implement your own allocator without having it conform to the Rust model, as far as I'm aware
  -# (This is only an issue because it explicitly keeps you from implementing some safety features I was planning to add)
• Custom targets are allowed but depend on the compiler version / are a nightly feature
• • Actually a lot of things are nightly features which isn't good
• The philosophy is so monolithic and non-modular it genuinely hurts
• For some reason the object files are way bigger than they need to be
• A lot of the safety features aren't actually available on no_std targets
• x86_64-unknown-none and x86_64-unknown-uefi are tier 2 targets, meaning they aren't guaranteed to pass tests
• Implicit allocations really aren't a good idea in general

[wary pecan]

i don't hate Rust for the record