A very small (not at all complete) x86_64 emulator written in Rust, whose sole goal is to run my OS.
It currently boots into some GRUB code.
BIOS interrupts currently emulated directly in rust, will move to assembly later.
No support for memory mapped registers, I/O ports, etc yet
#x86emu: From scratch x86 (maybe _64) emulator to run my OS on
1 messages · Page 1 of 1 (latest)
i wish you the best of luch with this
having anything close to the entire base x86-64 instruction set is a massive undertaking
I don't plan to get even close to this.. just being able to boot my OS with some old/simple hardware would already be enough
Aaand just encountered the first out instruction (GRUB, already in protected mode). Now it'll get interesting..
the test386.asm testsuite already passed for all of these steps:
| 0x00 | Real mode initialisation |
| 0x01 | Conditional jumps and loops |
| 0x02 | Quick tests of unsigned 32-bit multiplication and division |
| 0x03 | Move segment registers in real mode |
| 0x04 | Store, move, scan, and compare string data in real mode |
| 0x05 | Calls in real mode |
| 0x06 | Load full pointer in real mode |
currently stuck at ltr (not implemented yet)
neat
GRUB is doing some grub_reed_solomon_recover redundancy/error checking... this takes ages to execute
finally something interesting
GRUB successfully read the BIOS memory map from the emulater.. Now it wants to get the supported processor features. No idea, what I should support though
GRUB loaded successfully! (takes 3s to boot to this step right now)
We got GRUB version number 
it's getting shape
that was way easier than I thought it would be
is it x86-64 though?
i mean it also may be easy to get one particular piece of software working, but i am sure that if i were to try to e.g. run Windows 95 on that emulator, it wouldn't work :p
it support a bit of 64 already, but GRUB doesn't use it.. we'll see, when GRUB gives control to the OS.
probably the main point is BIOS/hardware inaccuracies, the CPU emulator itself hasn't caused a lot of problems (yet)
GRUB doesn't switch to 64-bit itself though, unless you're doing EFI and using the weird multiboot2 mode where it basically does glorified chainloading
but i assume this is BIOS, else you'd already be in 64-bit mode
yes, but my OS will
keep in mind you will need the entirety of x87, MMX, SSE, and SSE2 for real x86-64 support
only if I want to support everything.. my OS doesn't even use any FPU or SIMD instructions
this just took 20s to check... lol
why does this just work??? what???
never used Limine, no idea how to set it up
just create an image from the basic template at https://codeberg.org/Limine/limine-c-template-x86-64
basically download it and run make clean all or make clean all-hdd
former generates and ISO, latter HDD image
ok, I test it now
awesome, thanks
can I just emulate the hdd image?
what do you mean?
does it need any config?
lol... wait a minute
welp
is that cpuid?
idr my opcodes off the top of my head
but it looks a bit familiar
alrighty
yeah, basically that
Limine will definitely trigger more instructions like these since it expects i686 at minimum, while i think GRUB expects i386
it's mostly C
the bootsector part is in assembly, so are some routines somewhere, but the bulk of it is C
what's wrong with rdtsc? lol
I don't have MSRs yet
I hope it doesn't expect a realistic value here... I just return something that resembles ticks
should be fine
and now it wants to know the frequency through cpuid..
ah also, i forgot to mention that you can put serial: yes in the config for serial output
instructions aren't fixed-rate
don't have serial yet, only BIOS text mode 
uh, i don't think there is any code path in Limine that relies on knowing the CPU frequency?
what leaf is it?
add graphics: no to the config for that
though it should already automatically fall back to text mode if VBE isn't working/available
wait.. I said in cpuid that I support up to 0x16... maybe I should lower this
yes, that worked... not asking for frequency anymore
it just requested a memory map
awesome
and asking for drive paramers...
⭐
I think it's 486 in practice?
Even though officially they say 386
I meant for grub
I have this
thanks!
i'll star it whenever i log into my gitlab which i haven't used for many years lol
limine is diong lots of cmov.. I have to implement them all now
(obligatory consider codeberg instead)
yeah that's i686
damn, you work fast
wants to know how much RAM this has... well, idk..
why did it just downscale the video mode?
is 0xE0000 some kind of special address for limine? it tries to write there, but I don't have RAM at that address
wait.. it reads from there:
thread 'main' panicked at src/mem.rs:181:9:
not yet implemented: invalid memory access at 0xe0000. handle this?
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
CURRENT CPU STATE:
CS:IP = 0018:0000000000013EC0
SS:SP = 0020:000000000000EF6C
FLAGS = 0000000000000044
RAX = 00000000FFFFFFAE RBX = 00000000000E0000
RCX = 0000000000000008 RDX = 0000000000000003
RDI = 00000000000E0000 RSI = 000000000003A25A
RSP = 000000000000EF6C RBP = 000000000000EF90
CR0 = 0000000000000001 CR1 = 0000000000000000
CR2 = 0000000000000000 CR3 = 0000000000000000
CR4 = 0000000000000000 CR5 = 0000000000000000
CR6 = 0000000000000000 CR7 = 0000000000000000
CR8 = 0000000000000000
ES = 0020 base=00000000 limit=000FFFFF [AW...P.BG] dpl=0
CS = 0018 base=00000000 limit=000FFFFF [AR.E.P.DG] dpl=0
SS = 0020 base=00000000 limit=000FFFFF [AW...P.BG] dpl=0
DS = 0020 base=00000000 limit=000FFFFF [AW...P.BG] dpl=0
FS = 0020 base=00000000 limit=000FFFFF [AW...P.BG] dpl=0
GS = 0020 base=00000000 limit=000FFFFF [AW...P.BG] dpl=0
CPL = 0
32 bit protected mode
LAST INSTRUCTIONS:
[-] push esi
[-] call 0xffff9414 (=> 0x13eb1)
[-] push esi
[-] push edi
[-] mov edi, dword [0*esp+esp+0xc]
[-] mov esi, dword [0*esp+esp+0x10]
[-] mov ecx, dword [0*esp+esp+0x14]
[-] repz cmpsb
[-] cmp byte 0x52, byte 0x00
[-] a32 cmpsb
[-] jz 0x0b (=> 0x13ec3) (not taken)
[-] mov al, byte [edi-0x1]
[-] sub al, byte [esi-0x1]
[-] movsx eax, al
[-] jmp 0x02 [=> 0x13ed0]
[-] pop edi
[-] pop esi
[-] ret dword, 0
[-] add esp, dword 0x0000000c
[-] test eax, eax
[-] jnz 0xdc (=> 0x1aa80) (taken)
[-] cmp esi, dword 0x000ffff0
[-] lea esi, dword [esi+0x10]
[-] jnc 0x70 (=> 0x1aa8b) (not taken)
[-] cmp esi, edi
[-] cmovz esi, ebx
[-] push byte 0x08
[-] push dword 0x0003a25a
[-] push esi
[-] call 0xffff9414 (=> 0x13eb1)
[-] push esi
[-] push edi
[-] mov edi, dword [0*esp+esp+0xc]
[-] mov esi, dword [0*esp+esp+0x10]
[-] mov ecx, dword [0*esp+esp+0x14]
[-] repz cmpsb
I found it..
time for acpi man
I don't think it's required. I just have to handle unmapped accesses. Or map this region somewhere..
looks like ROM area
what is a bootloader even doing with ACPI?
passing it to the os
you don't have to find it yourself?
nooo
Limine just finds RSDP for the kernel
so kernel can just use limine_rsdp_request structure to get it
my OS searches it by itself.. because Multiboot 1 
AAAAH the auto-complete
it tries to access VGA registers... why can't you just use the BIOS??
and now it writes to VGA memory...
I don't know, if this is how this bootloader is supposed to work, but it prints this and then waits for input
but Emacs keybinds work in the GRUB editor.. I guess, I'll just stick with GRUB
yeah that's not how it is supposed to work lol
i mean, does it though?
i think if it does, it is just to disable the cursor
only 1 access.. the rest is VRAM.. which I implemented now, but apparently not correctly
the reason it uses VGA directly instead of BIOS to disable the cursor is that uh.... so like
you know in the real world
often firmware doesn't work
lol
one is reliable, the other isn't
how did you implement VRAM?
until now I had an array of bytes in the Video structure.. just changed accesses to this to Memory accesses.. So BIOS print now also writes in the video memory (address determined from mode) and display (called after BIOS print and periodically, as well as on hlt/crash) outputs this buffer (converting to UTF8 and printing, no colors yet)
GRUB uses the BIOS to disable the cursor...
look, there must be a reason, idr it off the top of my head
i'd need to check the blame if it was written there
I saw something ending with wn... or similar at the end of the video buffer (for some reason not rendered?). I'll look into this tomorrow, but the bug is probably just the rendering
alright take your time
you've done good work so far
sadly the commit doesn't really give a rationale
I need to clean up the instruction parsing a bit. Maybe declarative definitions and automate the rest.. this would also make adding a asm REPL later more straightforward
or I could implement gdb support
lol.. I found the bug... I casted an address to u16 before dividing it by 0x10 to convert it to a segment register.. so it got truncated
now I get something very different
(although it doesn't count down for some reason)
and I don't suppoort colors yet
but this looks like a bootloader
the next thing it wants are "shift flags", but I'll wait with implementing this, when I switched to SDL
that's awesome!
seems like my rdtsc returning the amount of fetched + executed instructions is not quite enough
and it boots fast, GRUB takes like 5s to get to the menu, Limine only ~500ms
neat!
no idea what to even optimize here
are you also writing the BIOS or just using SeaBIOS or similar?
custom "BIOS", but it's not assembly (yet), because I don't want to implement assembly disk-drivers just yet.. so I just grabbed an unused instruction to call back to the emulater, which emulates the BIOS calls
i see, interesting approach
This is the entire assembly BIOS stub right now
It's doing PCI now...
(GRUB)
let's see what GRUB does, if I just return 0xff for every PCI ADDRESS DATA port read
I think I found a bug in GRUB.. should this not be gdt_end - gdt - 1?
I thinks that incorrect
we can also ask @neat ibex
they are bootloader magicians
this crashed my emulator, because I asserted that the size is a multiple of 8..
ehm... why did GRUB halt the CPU?
waaaait... this isn't GRUB...
this is MY OS???
what???
but I print to a fixed buffer.... which is not the one GRUB gave me... already found bug in OS
but GRUB just booted it??
but it takes like 1min to boot through GRUB
2 minutes*
well, I don't really know what to optimize.. most time is taken by fetch_byte, but nothing in there really takes time...
see #1433851469827801139 message
maaaybe add prefetch
I thought... but would this really help?
instead of loading one instruction you load n-bytes
fetch_byte() is only RAM access with 2 indirections and a few cmps
and it adds up quickly doesnt it
but GRUB is doing lots of small reads from disk... maybe there is a bug in my BIOS 0x13.. and GRUB just does more?
forgot to recompile bootloader... another 2 minutes wasted
3 minutes
aaand real message from the OS
seems like, I have to implement something...
maybe I could pre-fetch the maximum instruction length? But I'd have to prevent faults somehow
I added prefetch... it increased execution time
nice instruction cache...
hows the emulator twin
I was continuing the OS this week, now it doesn't boot anymore in the emulator:
inspired= true;
I want to write one too now lol
anyone know if 0xf00000 is some kind of special address?
That would be where the ISA memory hole is
so am I supposed to have some MMIO here?
but GRUB shouldn't just access random memory..
True -- I assume your emulator uses BIOS and that you've written the firmware yourself, did you implement the memory map probing routines by any chance?
because I know that GRUB will start using the first block marked as "free" from that e820 subroutine call, I can't remember how much the implementation wants in terms of size though
I did.. and the "BIOS" is just an unused instruction calling directly into the emulator.. So I didn't port SeaBIOS or similar yet.. and the firmware is extremely high-level as most devices don't actually exist yet.
the map:
[BIOS] load memory map to 6800:0004 (cont=00000000)
[BIOS] 0000000000000000: 0000000000000500 bytes, type 2
[BIOS] load memory map to 6800:0004 (cont=00000001)
[BIOS] 0000000000000500: 0000000000080000 bytes, type 1
[BIOS] load memory map to 6800:0004 (cont=00000002)
[BIOS] 0000000000080000: 00000000000A0000 bytes, type 2
[BIOS] load memory map to 6800:0004 (cont=00000003)
[BIOS] 0000000000100000: 0000000000E00000 bytes, type 1
[BIOS] load memory map to 6800:0004 (cont=00000004)
[BIOS] 0000000001000000: 000000000F000000 bytes, type 1
[BIOS] load memory map to 6800:0004 (cont=00000005)
[BIOS] 00000000000A0000: 00000000000C0000 bytes, type 2
[BIOS] load memory map to 6800:0004 (cont=00000006)
[BIOS] 00000000000F0000: 0000000000010000 bytes, type 2
[BIOS] load memory map to 6800:0004 (cont=00000007)
[BIOS] 00000000000C8000: 0000000000028000 bytes, type 2
[BIOS] load memory map to 6800:0004 (cont=00000008)
[BIOS] 00000000FFFF0000: 0000000000010000 bytes, type 2
[BIOS] load memory map to 6800:0004 (cont=00000000)
[BIOS] 0000000000000000: 0000000000000500 bytes, type 2
[BIOS] load memory map to 6800:0004 (cont=00000001)
[BIOS] 0000000000000500: 0000000000080000 bytes, type 1
[BIOS] load memory map to 6800:0004 (cont=00000002)
[BIOS] 0000000000080000: 00000000000A0000 bytes, type 2
[BIOS] load memory map to 6800:0004 (cont=00000003)
[BIOS] 0000000000100000: 0000000000E00000 bytes, type 1
[BIOS] load memory map to 6800:0004 (cont=00000004)
[BIOS] 0000000001000000: 000000000F000000 bytes, type 1
[BIOS] load memory map to 6800:0004 (cont=00000005)
[BIOS] 00000000000A0000: 00000000000C0000 bytes, type 2
[BIOS] load memory map to 6800:0004 (cont=00000006)
[BIOS] 00000000000F0000: 0000000000010000 bytes, type 2
[BIOS] load memory map to 6800:0004 (cont=00000007)
[BIOS] 00000000000C8000: 0000000000028000 bytes, type 2
[BIOS] load memory map to 6800:0004 (cont=00000008)
[BIOS] 00000000FFFF0000: 0000000000010000 bytes, type 2
Yeah I was expecting that answer too to be honest lol
Pretty similar to what DOSBox does, but still different because this is x86_64.. so that's cool!
[BIOS] load memory map to 6800:0004 (cont=00000006)
[BIOS] 00000000000F0000: 0000000000010000 bytes, type 2
huh? why am I setting this... wait a minute...
and yes I was about to say this lol
Isn't these regions overlapping with each-other too?
[BIOS] load memory map to 6800:0004 (cont=00000005)
[BIOS] 00000000000A0000: 00000000000C0000 bytes, type 2
[BIOS] load memory map to 6800:0004 (cont=00000006)
[BIOS] 00000000000F0000: 0000000000010000 bytes, type 2
[BIOS] load memory map to 6800:0004 (cont=00000007)
[BIOS] 00000000000C8000: 0000000000028000 bytes, type 2
well, they are.. idk why
the interrupt code:
0x20 => 'map: { // get system memory map
let eax = self.regs[AX as usize].r32();
let edx = self.regs[DX as usize].r32();
let ebx = self.regs[BX as usize].r32();
let ecx = self.regs[CX as usize].r32();
let es = SegReg::Extra.reg(self).reg.r16();
let di = self.regs[DI as usize].r16();
if eax != 0x0000E820 || edx != 0x534D4150 /* SMAP */ {
eprintln!("[BIOS] int 0x15 ax=e820: invalid eax/edx: {eax:08X}/{edx:08X}");
break 'map;
}
eprintln!("[BIOS] load memory map to {es:04X}:{di:04X} (cont={ebx:08X})");
assert!(ecx >= 20);
if let Some(&Region {start, len, reserved, ..}) = m.mem.regions.get(ebx as usize) {
let typ = reserved as u64 + 1;
m.mem.set_sized( di as u64, RegSize::R64, self.es(), Reg::new(start));
m.mem.set_sized((di + 0x08) as u64, RegSize::R64, self.es(), Reg::new(len as u64));
m.mem.set_sized((di + 0x10) as u64, RegSize::R32, self.es(), Reg::new(typ));
eprintln!("[BIOS] {start:016X}: {len:016X} bytes, type {}", typ);
flags &= !CF as u16;
self.regs[AX as usize].r32s(0x534D4150); // SMAP
self.regs[CX as usize].r32s(20);
self.regs[BX as usize].r32s(if ebx as usize + 1 >= m.mem.regions.len() {
0
} else {
ebx + 1
});
} else {
eprintln!("[BIOS] invalid continuation number {ebx} for memory map");
flags |= CF as u16;
*self.regs[AX as usize].him() = 0x86;
}
},
m.mem.regions should contain all regions..
that's what I map:
m.map_ram(0x00000000, 0x000A0000,
&[(true, 0x00000000, 0x00000500), // IVT, BIOS data
(false, 0x00000500, 0x00080000), // low memory
(true, 0x00080000, 0x000A0000), // EBDA
]);
m.map_ram(0x00100000, 0x00F00000, &[]); // extended memory
m.map_ram(0x01000000, 0x10000000, &[]); // more extended memory
m.map_ram(0x000A0000, 0x000C0000, &[(true, 0x000A0000, 0x000C0000)]); // VIDEO RAM
I think you got a positional issue in your Some(&Region {start, len, reserved ..}]
I think len actually contains end, based on what you gave me
That or something similar when you create those structures
pub struct Region {
pub start: u64,
pub len: usize,
pub reserved: bool,
}
rust destructuring is based on names. if you don't do start: name or whatever. so they are 100% correct
everything is correct except those 2:
[BIOS] load memory map to 6800:0004 (cont=00000006)
[BIOS] 00000000000F0000: 0000000000010000 bytes, type 2
[BIOS] load memory map to 6800:0004 (cont=00000007)
[BIOS] 00000000000C8000: 0000000000028000 bytes, type 2
wait, no... F0000 is the BIOS shadow area, that's correct too
the fault is at F00000 not F0000
F00000 is just past the end of this region:
[BIOS] load memory map to 6800:0004 (cont=00000003)
[BIOS] 0000000000100000: 0000000000E00000 bytes, type 1
Is the end address inclusive or exclusive?
that's the length, not the end address
so exclusive
I'm trying to capture the instruction right now, this could take a while...
Good luck lol
finished. it's the rep movsb reading or writing out-of-bounds for some reason
[-] a32 movsb
[-] a32 movsb
[-] a32 movsb
[-] a32 movsb
[-] a32 movsb
[-] a32 movsb
[-] a32 movsb
[-] a32 movsb
thread 'main' panicked at src/mem.rs:202:33:
not yet implemented: unmapped memory access at 0xf00000. handle this?
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
cpu halted. executed 269249450 instructions
CURRENT CPU STATE:
CS:IP = 0008:0000000000100030
SS:SP = 0010:000000000007FC80
FLAGS = 0000000000000054
RAX = 0000000001000000 RBX = 0000000000000000
RCX = 0000000001EBEF08 RDX = 0000000000E921F0
RDI = 0000000000F00000 RSI = 0000000001D00000
RSP = 000000000007FC80 RBP = 000000000007FCAC
CR0 = 0000000000000001 CR1 = 0000000000000000
CR2 = 0000000000000000 CR3 = 0000000000000000
CR4 = 0000000000000000 CR5 = 0000000000000000
CR6 = 0000000000000000 CR7 = 0000000000000000
CR8 = 0000000000000000
ES = 0010 base=00000000 limit=000FFFFF [.W...P.BG] dpl=0
CS = 0008 base=00000000 limit=000FFFFF [.R.E.P.DG] dpl=0
SS = 0010 base=00000000 limit=000FFFFF [.W...P.BG] dpl=0
DS = 0010 base=00000000 limit=000FFFFF [.W...P.BG] dpl=0
FS = 0010 base=00000000 limit=000FFFFF [.W...P.BG] dpl=0
GS = 0010 base=00000000 limit=000FFFFF [.W...P.BG] dpl=0
CPL = 0
32 bit protected mode
after executing 200 million instructions..
RDI is out of bounds and still 0x1EBEF08 bytes left
I just hit a random key on my keyboard lol
That's a lot of bytes for movsb
30 megs wow
that's only after the fault though
they are a lot more
but this looks correct... probably GRUB loading my kernel which is 24MiB...
Is your kernel loaded at 0x1000000?
this is the last rep movsb
[-] rep movsb 10:200000, 10:1000000, 2BBEF08
and ignore the segments, the base is 0 on all of them
ES = 0010 base=00000000 limit=000FFFFF [.W...P.BG] dpl=0
CS = 0008 base=00000000 limit=000FFFFF [.R.E.P.DG] dpl=0
SS = 0010 base=00000000 limit=000FFFFF [.W...P.BG] dpl=0
DS = 0010 base=00000000 limit=000FFFFF [.W...P.BG] dpl=0
FS = 0010 base=00000000 limit=000FFFFF [.W...P.BG] dpl=0
GS = 0010 base=00000000 limit=000FFFFF [.W...P.BG] dpl=0
the kernel is only 0x17F4BE8 bytes..
it could still be a bug in one of my instructions causing the ecx register to have the wrong value...
What's weird also is that RIP points at 0x100030, which is near the start of GRUB's main binary if I'm not mistaken
I would expect it to be a bit further, especially if you were able to pop up the boot selection menu and the command line
it's probably the copy code, which jumps to a small stub to be able to overwrite itself or sth?
If GRUB does a far jmp at the very start of the binary, yes
I'm checking if they do something like this
this would fit perfectly.. but just the address in $esi is not the one I get
is it able to run your os now?
no... still getting that weird invalid access