#x86emu: From scratch x86 (maybe _64) emulator to run my OS on

1 messages · Page 1 of 1 (latest)

late reef
#

A very small (not at all complete) x86_64 emulator written in Rust, whose sole goal is to run my OS.
It currently boots into some GRUB code.
BIOS interrupts currently emulated directly in rust, will move to assembly later.
No support for memory mapped registers, I/O ports, etc yet

#

I don't think that's correct... GRUB is jumping to the null segment!?

neat ibex
#

i wish you the best of luch with this

#

having anything close to the entire base x86-64 instruction set is a massive undertaking

late reef
#

Aaand just encountered the first out instruction (GRUB, already in protected mode). Now it'll get interesting..

late reef
# neat ibex having anything close to the entire base x86-64 instruction set is a massive und...

the test386.asm testsuite already passed for all of these steps:

| 0x00 | Real mode initialisation                                           |
| 0x01 | Conditional jumps and loops                                        |
| 0x02 | Quick tests of unsigned 32-bit multiplication and division         |
| 0x03 | Move segment registers in real mode                                |
| 0x04 | Store, move, scan, and compare string data in real mode            |
| 0x05 | Calls in real mode                                                 |
| 0x06 | Load full pointer in real mode                                     |

currently stuck at ltr (not implemented yet)

neat ibex
#

neat

late reef
#

GRUB is doing some grub_reed_solomon_recover redundancy/error checking... this takes ages to execute

late reef
#

finally something interesting

late reef
#

GRUB successfully read the BIOS memory map from the emulater.. Now it wants to get the supported processor features. No idea, what I should support though

late reef
#

GRUB loaded successfully! (takes 3s to boot to this step right now)

late reef
#

We got GRUB version number KEKW

late reef
#

it's getting shape

late reef
neat ibex
#

is it x86-64 though?

#

i mean it also may be easy to get one particular piece of software working, but i am sure that if i were to try to e.g. run Windows 95 on that emulator, it wouldn't work :p

late reef
late reef
neat ibex
#

but i assume this is BIOS, else you'd already be in 64-bit mode

neat ibex
#

keep in mind you will need the entirety of x87, MMX, SSE, and SSE2 for real x86-64 support

late reef
#

only if I want to support everything.. my OS doesn't even use any FPU or SIMD instructions

#

this just took 20s to check... lol

neat ibex
#

try Limine

#

out of curiosity

late reef
#

why does this just work??? what???

late reef
neat ibex
#

basically download it and run make clean all or make clean all-hdd

#

former generates and ISO, latter HDD image

late reef
#

this requires sgdisk

#

ah... found it

neat ibex
#

install it, it's present in pretty much every distro

#

super portable tool

late reef
#

ok, I test it now

neat ibex
#

awesome, thanks

late reef
#

can I just emulate the hdd image?

neat ibex
#

what do you mean?

late reef
#

does it need any config?

neat ibex
#

nah, just run the image

#

it has a config but it's already inside the image

late reef
#

lol... wait a minute

neat ibex
#

welp

#

is that cpuid?

#

idr my opcodes off the top of my head

#

but it looks a bit familiar

late reef
#

bts instruction

#

should not be too hard

#

I say, when it runs

neat ibex
#

alrighty

late reef
#

that's just bt but it also sets the bit afterwards..

#

let's see..

neat ibex
#

yeah, basically that

late reef
#

next instruction: setnc

#

is this written in assembly?

neat ibex
#

Limine will definitely trigger more instructions like these since it expects i686 at minimum, while i think GRUB expects i386

neat ibex
late reef
#

these do not feel like gcc-emitted instruction

#

rdtsc.. lol... wtf...

neat ibex
#

the bootsector part is in assembly, so are some routines somewhere, but the bulk of it is C

#

what's wrong with rdtsc? lol

late reef
#

I don't have MSRs yet

late reef
neat ibex
#

should be fine

late reef
#

and now it wants to know the frequency through cpuid..

neat ibex
#

ah also, i forgot to mention that you can put serial: yes in the config for serial output

late reef
#

instructions aren't fixed-rate

late reef
neat ibex
#

uh, i don't think there is any code path in Limine that relies on knowing the CPU frequency?

#

what leaf is it?

neat ibex
#

though it should already automatically fall back to text mode if VBE isn't working/available

late reef
#

wait.. I said in cpuid that I support up to 0x16... maybe I should lower this

#

yes, that worked... not asking for frequency anymore

#

it just requested a memory map

neat ibex
#

awesome

late reef
#

and asking for drive paramers...

finite otter
#

do u have github repo for this?

#

i wanna star

shadow mulch
#

sharp knot
#

Even though officially they say 386

neat ibex
#

it's 686

#

oh you mean GRUB

sharp knot
#

I meant for grub

neat ibex
#

idk for sure

#

either 386 or 486

finite otter
#

Thank you

#

I will star it

#

Deserved

#

Starred

late reef
#

thanks!

neat ibex
#

i'll star it whenever i log into my gitlab which i haven't used for many years lol

late reef
#

limine is diong lots of cmov.. I have to implement them all now

neat ibex
#

(obligatory consider codeberg instead)

late reef
#

index out of bounds, lol

#

finally... it moved the cursor

#

we could see somthing soon

neat ibex
#

damn, you work fast

late reef
#

wants to know how much RAM this has... well, idk..

#

why did it just downscale the video mode?

#

is 0xE0000 some kind of special address for limine? it tries to write there, but I don't have RAM at that address

#

wait.. it reads from there:

thread 'main' panicked at src/mem.rs:181:9:
not yet implemented: invalid memory access at 0xe0000. handle this?
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
CURRENT CPU STATE:
CS:IP = 0018:0000000000013EC0
SS:SP = 0020:000000000000EF6C
FLAGS = 0000000000000044

RAX = 00000000FFFFFFAE  RBX = 00000000000E0000
RCX = 0000000000000008  RDX = 0000000000000003
RDI = 00000000000E0000  RSI = 000000000003A25A
RSP = 000000000000EF6C  RBP = 000000000000EF90

CR0 = 0000000000000001  CR1 = 0000000000000000
CR2 = 0000000000000000  CR3 = 0000000000000000
CR4 = 0000000000000000  CR5 = 0000000000000000
CR6 = 0000000000000000  CR7 = 0000000000000000
CR8 = 0000000000000000

ES = 0020 base=00000000 limit=000FFFFF [AW...P.BG] dpl=0
CS = 0018 base=00000000 limit=000FFFFF [AR.E.P.DG] dpl=0
SS = 0020 base=00000000 limit=000FFFFF [AW...P.BG] dpl=0
DS = 0020 base=00000000 limit=000FFFFF [AW...P.BG] dpl=0
FS = 0020 base=00000000 limit=000FFFFF [AW...P.BG] dpl=0
GS = 0020 base=00000000 limit=000FFFFF [AW...P.BG] dpl=0

CPL = 0
32 bit protected mode

LAST INSTRUCTIONS:
[-] push esi
[-] call 0xffff9414 (=> 0x13eb1)
[-] push esi
[-] push edi
[-] mov edi, dword [0*esp+esp+0xc]
[-] mov esi, dword [0*esp+esp+0x10]
[-] mov ecx, dword [0*esp+esp+0x14]
[-] repz cmpsb
[-] cmp byte 0x52, byte 0x00
[-] a32 cmpsb
[-] jz 0x0b (=> 0x13ec3) (not taken)
[-] mov al, byte [edi-0x1]
[-] sub al, byte [esi-0x1]
[-] movsx eax, al
[-] jmp 0x02 [=> 0x13ed0]
[-] pop edi
[-] pop esi
[-] ret dword, 0
[-] add esp, dword 0x0000000c
[-] test eax, eax
[-] jnz 0xdc (=> 0x1aa80) (taken)
[-] cmp esi, dword 0x000ffff0
[-] lea esi, dword [esi+0x10]
[-] jnc 0x70 (=> 0x1aa8b) (not taken)
[-] cmp esi, edi
[-] cmovz esi, ebx
[-] push byte 0x08
[-] push dword 0x0003a25a
[-] push esi
[-] call 0xffff9414 (=> 0x13eb1)
[-] push esi
[-] push edi
[-] mov edi, dword [0*esp+esp+0xc]
[-] mov esi, dword [0*esp+esp+0x10]
[-] mov ecx, dword [0*esp+esp+0x14]
[-] repz cmpsb
#

I found it..

finite otter
#

time for acpi man

late reef
#

looks like ROM area

late reef
sharp knot
#

passing it to the os

late reef
#

you don't have to find it yourself?

finite otter
#

nooo

#

Limine just finds RSDP for the kernel

#

so kernel can just use limine_rsdp_request structure to get it

late reef
#

my OS searches it by itself.. because Multiboot 1 KEKW

#

AAAAH the auto-complete

#

it tries to access VGA registers... why can't you just use the BIOS??

#

and now it writes to VGA memory...

late reef
#

I don't know, if this is how this bootloader is supposed to work, but it prints this and then waits for input

#

but Emacs keybinds work in the GRUB editor.. I guess, I'll just stick with GRUB

neat ibex
#

yeah that's not how it is supposed to work lol

neat ibex
#

i think if it does, it is just to disable the cursor

late reef
neat ibex
#

the reason it uses VGA directly instead of BIOS to disable the cursor is that uh.... so like

#

you know in the real world

#

often firmware doesn't work

#

lol

#

one is reliable, the other isn't

#

how did you implement VRAM?

late reef
# neat ibex how did you implement VRAM?

until now I had an array of bytes in the Video structure.. just changed accesses to this to Memory accesses.. So BIOS print now also writes in the video memory (address determined from mode) and display (called after BIOS print and periodically, as well as on hlt/crash) outputs this buffer (converting to UTF8 and printing, no colors yet)

late reef
neat ibex
#

look, there must be a reason, idr it off the top of my head

#

i'd need to check the blame if it was written there

late reef
#

I saw something ending with wn... or similar at the end of the video buffer (for some reason not rendered?). I'll look into this tomorrow, but the bug is probably just the rendering

neat ibex
#

alright take your time

#

you've done good work so far

#

sadly the commit doesn't really give a rationale

late reef
#

I need to clean up the instruction parsing a bit. Maybe declarative definitions and automate the rest.. this would also make adding a asm REPL later more straightforward

late reef
#

or I could implement gdb support

late reef
#

now I get something very different

#

(although it doesn't count down for some reason)

#

and I don't suppoort colors yet

#

but this looks like a bootloader

#

the next thing it wants are "shift flags", but I'll wait with implementing this, when I switched to SDL

neat ibex
late reef
late reef
neat ibex
#

neat!

late reef
#

no idea what to even optimize here

neat ibex
#

are you also writing the BIOS or just using SeaBIOS or similar?

late reef
neat ibex
#

i see, interesting approach

late reef
#

It's doing PCI now...

#

(GRUB)

#

let's see what GRUB does, if I just return 0xff for every PCI ADDRESS DATA port read

finite otter
#

thats so awesome

late reef
finite otter
#

we can also ask @neat ibex

#

they are bootloader magicians

late reef
#

this crashed my emulator, because I asserted that the size is a multiple of 8..

#

ehm... why did GRUB halt the CPU?

#

waaaait... this isn't GRUB...

#

this is MY OS???

#

what???

#

but I print to a fixed buffer.... which is not the one GRUB gave me... already found bug in OS

#

but GRUB just booted it??

#

but it takes like 1min to boot through GRUB

#

2 minutes*

finite otter
#

very fast

#

but dont worry about that, you can always optimize

late reef
#

well, I don't really know what to optimize.. most time is taken by fetch_byte, but nothing in there really takes time...

late reef
finite otter
late reef
#

I thought... but would this really help?

finite otter
#

instead of loading one instruction you load n-bytes

late reef
#

fetch_byte() is only RAM access with 2 indirections and a few cmps

finite otter
#

and it adds up quickly doesnt it

late reef
#

but GRUB is doing lots of small reads from disk... maybe there is a bug in my BIOS 0x13.. and GRUB just does more?

#

forgot to recompile bootloader... another 2 minutes wasted

#

3 minutes

late reef
late reef
#

seems like, I have to implement something...

late reef
late reef
#

I added prefetch... it increased execution time

late reef
#

nice instruction cache...

finite otter
#

hows the emulator twin

late reef
snow dawn
#

inspired= true;
I want to write one too now lol

late reef
#

anyone know if 0xf00000 is some kind of special address?

late reef
#

but GRUB shouldn't just access random memory..

lavish siren
#

because I know that GRUB will start using the first block marked as "free" from that e820 subroutine call, I can't remember how much the implementation wants in terms of size though

late reef
#

the map:

[BIOS] load memory map to 6800:0004 (cont=00000000)
[BIOS]   0000000000000000: 0000000000000500 bytes, type 2
[BIOS] load memory map to 6800:0004 (cont=00000001)
[BIOS]   0000000000000500: 0000000000080000 bytes, type 1
[BIOS] load memory map to 6800:0004 (cont=00000002)
[BIOS]   0000000000080000: 00000000000A0000 bytes, type 2
[BIOS] load memory map to 6800:0004 (cont=00000003)
[BIOS]   0000000000100000: 0000000000E00000 bytes, type 1
[BIOS] load memory map to 6800:0004 (cont=00000004)
[BIOS]   0000000001000000: 000000000F000000 bytes, type 1
[BIOS] load memory map to 6800:0004 (cont=00000005)
[BIOS]   00000000000A0000: 00000000000C0000 bytes, type 2
[BIOS] load memory map to 6800:0004 (cont=00000006)
[BIOS]   00000000000F0000: 0000000000010000 bytes, type 2
[BIOS] load memory map to 6800:0004 (cont=00000007)
[BIOS]   00000000000C8000: 0000000000028000 bytes, type 2
[BIOS] load memory map to 6800:0004 (cont=00000008)
[BIOS]   00000000FFFF0000: 0000000000010000 bytes, type 2
[BIOS] load memory map to 6800:0004 (cont=00000000)
[BIOS]   0000000000000000: 0000000000000500 bytes, type 2
[BIOS] load memory map to 6800:0004 (cont=00000001)
[BIOS]   0000000000000500: 0000000000080000 bytes, type 1
[BIOS] load memory map to 6800:0004 (cont=00000002)
[BIOS]   0000000000080000: 00000000000A0000 bytes, type 2
[BIOS] load memory map to 6800:0004 (cont=00000003)
[BIOS]   0000000000100000: 0000000000E00000 bytes, type 1
[BIOS] load memory map to 6800:0004 (cont=00000004)
[BIOS]   0000000001000000: 000000000F000000 bytes, type 1
[BIOS] load memory map to 6800:0004 (cont=00000005)
[BIOS]   00000000000A0000: 00000000000C0000 bytes, type 2
[BIOS] load memory map to 6800:0004 (cont=00000006)
[BIOS]   00000000000F0000: 0000000000010000 bytes, type 2
[BIOS] load memory map to 6800:0004 (cont=00000007)
[BIOS]   00000000000C8000: 0000000000028000 bytes, type 2
[BIOS] load memory map to 6800:0004 (cont=00000008)
[BIOS]   00000000FFFF0000: 0000000000010000 bytes, type 2
lavish siren
late reef
#

[BIOS] load memory map to 6800:0004 (cont=00000006)
[BIOS] 00000000000F0000: 0000000000010000 bytes, type 2

huh? why am I setting this... wait a minute...

lavish siren
#

and yes I was about to say this lol

#

Isn't these regions overlapping with each-other too?

[BIOS] load memory map to 6800:0004 (cont=00000005)
[BIOS]   00000000000A0000: 00000000000C0000 bytes, type 2
[BIOS] load memory map to 6800:0004 (cont=00000006)
[BIOS]   00000000000F0000: 0000000000010000 bytes, type 2
[BIOS] load memory map to 6800:0004 (cont=00000007)
[BIOS]   00000000000C8000: 0000000000028000 bytes, type 2
late reef
#

the interrupt code:

0x20 => 'map: { // get system memory map
    let eax = self.regs[AX as usize].r32();
    let edx = self.regs[DX as usize].r32();

    let ebx = self.regs[BX as usize].r32();
    let ecx = self.regs[CX as usize].r32();

    let es = SegReg::Extra.reg(self).reg.r16();
    let di = self.regs[DI as usize].r16();

    if eax != 0x0000E820 || edx != 0x534D4150 /* SMAP */ {
        eprintln!("[BIOS] int 0x15 ax=e820: invalid eax/edx: {eax:08X}/{edx:08X}");
        break 'map;
    }

    eprintln!("[BIOS] load memory map to {es:04X}:{di:04X} (cont={ebx:08X})");
    assert!(ecx >= 20);

    if let Some(&Region {start, len, reserved, ..}) = m.mem.regions.get(ebx as usize) {
        let typ = reserved as u64 + 1;
        m.mem.set_sized( di         as u64, RegSize::R64, self.es(), Reg::new(start));
        m.mem.set_sized((di + 0x08) as u64, RegSize::R64, self.es(), Reg::new(len as u64));
        m.mem.set_sized((di + 0x10) as u64, RegSize::R32, self.es(), Reg::new(typ));

        eprintln!("[BIOS]   {start:016X}: {len:016X} bytes, type {}", typ);

        flags &= !CF as u16;
        self.regs[AX as usize].r32s(0x534D4150); // SMAP
        self.regs[CX as usize].r32s(20);
        self.regs[BX as usize].r32s(if ebx as usize + 1 >= m.mem.regions.len() {
            0
        } else {
            ebx + 1
        });
    } else {
        eprintln!("[BIOS] invalid continuation number {ebx} for memory map");
        flags |= CF as u16;
        *self.regs[AX as usize].him() = 0x86;
    }
},
#

m.mem.regions should contain all regions..

#

that's what I map:

m.map_ram(0x00000000, 0x000A0000,
          &[(true,  0x00000000, 0x00000500), // IVT, BIOS data
            (false, 0x00000500, 0x00080000), // low memory
            (true,  0x00080000, 0x000A0000), // EBDA
          ]);
                                                                                   
m.map_ram(0x00100000, 0x00F00000, &[]); // extended memory
m.map_ram(0x01000000, 0x10000000, &[]); // more extended memory
m.map_ram(0x000A0000, 0x000C0000, &[(true, 0x000A0000, 0x000C0000)]); // VIDEO RAM
lavish siren
#

I think you got a positional issue in your Some(&Region {start, len, reserved ..}]

#

I think len actually contains end, based on what you gave me

#

That or something similar when you create those structures

late reef
#
pub struct Region {
    pub start: u64,
    pub len: usize,
    pub reserved: bool,
}

rust destructuring is based on names. if you don't do start: name or whatever. so they are 100% correct

#

everything is correct except those 2:

[BIOS] load memory map to 6800:0004 (cont=00000006)
[BIOS]   00000000000F0000: 0000000000010000 bytes, type 2
[BIOS] load memory map to 6800:0004 (cont=00000007)
[BIOS]   00000000000C8000: 0000000000028000 bytes, type 2
#

wait, no... F0000 is the BIOS shadow area, that's correct too

#

the fault is at F00000 not F0000

#

F00000 is just past the end of this region:

[BIOS] load memory map to 6800:0004 (cont=00000003)
[BIOS]   0000000000100000: 0000000000E00000 bytes, type 1
lavish siren
late reef
#

that's the length, not the end address

#

so exclusive

#

I'm trying to capture the instruction right now, this could take a while...

late reef
#

bruh... I need to optimize this..

#

just because interrupts might happen...

late reef
#
[-] a32 movsb
[-] a32 movsb
[-] a32 movsb
[-] a32 movsb
[-] a32 movsb
[-] a32 movsb
[-] a32 movsb
[-] a32 movsb

thread 'main' panicked at src/mem.rs:202:33:
not yet implemented: unmapped memory access at 0xf00000. handle this?
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
cpu halted. executed 269249450 instructions
CURRENT CPU STATE:
CS:IP = 0008:0000000000100030
SS:SP = 0010:000000000007FC80
FLAGS = 0000000000000054

RAX = 0000000001000000  RBX = 0000000000000000
RCX = 0000000001EBEF08  RDX = 0000000000E921F0
RDI = 0000000000F00000  RSI = 0000000001D00000
RSP = 000000000007FC80  RBP = 000000000007FCAC

CR0 = 0000000000000001  CR1 = 0000000000000000
CR2 = 0000000000000000  CR3 = 0000000000000000
CR4 = 0000000000000000  CR5 = 0000000000000000
CR6 = 0000000000000000  CR7 = 0000000000000000
CR8 = 0000000000000000

ES = 0010 base=00000000 limit=000FFFFF [.W...P.BG] dpl=0
CS = 0008 base=00000000 limit=000FFFFF [.R.E.P.DG] dpl=0
SS = 0010 base=00000000 limit=000FFFFF [.W...P.BG] dpl=0
DS = 0010 base=00000000 limit=000FFFFF [.W...P.BG] dpl=0
FS = 0010 base=00000000 limit=000FFFFF [.W...P.BG] dpl=0
GS = 0010 base=00000000 limit=000FFFFF [.W...P.BG] dpl=0

CPL = 0
32 bit protected mode

after executing 200 million instructions..

#

RDI is out of bounds and still 0x1EBEF08 bytes left

lavish siren
#

I just hit a random key on my keyboard lol

#

That's a lot of bytes for movsb

#

30 megs wow

late reef
#

that's only after the fault though

#

they are a lot more

#

but this looks correct... probably GRUB loading my kernel which is 24MiB...

lavish siren
#

Is your kernel loaded at 0x1000000?

late reef
#

this is the last rep movsb
[-] rep movsb 10:200000, 10:1000000, 2BBEF08

late reef
#

43 MiB, lol

lavish siren
#

2BBEF08 vs 1EBEF08

late reef
#

and ignore the segments, the base is 0 on all of them

#
ES = 0010 base=00000000 limit=000FFFFF [.W...P.BG] dpl=0
CS = 0008 base=00000000 limit=000FFFFF [.R.E.P.DG] dpl=0
SS = 0010 base=00000000 limit=000FFFFF [.W...P.BG] dpl=0
DS = 0010 base=00000000 limit=000FFFFF [.W...P.BG] dpl=0
FS = 0010 base=00000000 limit=000FFFFF [.W...P.BG] dpl=0
GS = 0010 base=00000000 limit=000FFFFF [.W...P.BG] dpl=0
lavish siren
#

Yeah I did see them

#

I know GRUB makes everything flat anyway but thanks lol

late reef
#

the kernel is only 0x17F4BE8 bytes..

#

it could still be a bug in one of my instructions causing the ecx register to have the wrong value...

lavish siren
#

What's weird also is that RIP points at 0x100030, which is near the start of GRUB's main binary if I'm not mistaken

#

I would expect it to be a bit further, especially if you were able to pop up the boot selection menu and the command line

late reef
#

it's probably the copy code, which jumps to a small stub to be able to overwrite itself or sth?

lavish siren
late reef
#

this would fit perfectly.. but just the address in $esi is not the one I get

plush socket
#

is it able to run your os now?

late reef