anyway, that could be solved just by doing smaller reads and stitching the bytes together
#FrostyOS
lost schooner
dusky orbit
imo its always worth being as spec-compliant as possible
just so you can be sure the compiler is doing what you expect
lost schooner
I removed all the unnecessary disabling of ubsan
still the same outcome
works with -O1, but not -O2
dusky orbit
any reason you dont log anything from ubsan?
you could try directing all of them to a single stub function and break there, looking at the callstack would reveal where you're getting UB
lost schooner
any reason you dont log anything from ubsan?
what is there to log? Any violation results in a kernel panic.
dusky orbit
gotcha, well why not follow the trace from the panic?
that should lead you to where the UB happened
lost schooner
I don't get any panics
dusky orbit
hmm
lost schooner
no ubsan violations
its just messing something up
not that this will mean anything, but running with -O3 has the same result as -O2 when ubsan is enabled (idk about without ubsan).
so there is definitely some UB somewhere
as for where, I have no clue
I might try single stepping through some stuff with gdb
lost schooner
i single stepped through all the I/O APIC init, and couldn't identify anything
with -O2:
and with -O1:
this immediately tells me that some bogus values are being written to the I/O APIC redirection entries
all fixed now
that was such a painful bug
thanks @dusky orbit for your help
hollow trail
what was the bug?
lost schooner
I was forgetting to clear the redirection entries when I first create them
hollow trail
oh
lost schooner
because I create the entries like this, ubsan was putting some bogus value in them:
x86_64_IOAPIC_RedirectionEntry entry;
there is also a possiblity that this is the issue I was having on real hardware
proud nova
I was forgetting to clear the redirection entries when I first create them
A classic
hollow trail
there is also a possiblity that this is the issue I was having on real hardware
give it a shot now
lost schooner
Thats what Im attempting right now. Ill even leave ubsan on
proud nova
Didn't know you could turn on ubsan for freestanding projects.
lost schooner
You need to have a custom implementation
Anyway, it doesnt work
Nothing is printed to the screen
hollow trail
damn
lost schooner
It must freeze during uACPI init though as quickly pressing the power button doesnt cause a shutdown
Its probably SMP related
I havent tested it with more than 2 cores
My test device has 4
proud nova
I can imaging going multicore can cause interesting problems
hollow trail
Try without smp
proud nova
How far along is your project? Have you run any userspace code so far?
lost schooner
I have userspace. I have just under 40 syscalls, many of which are similar to others. No relocation, and a very basic libc. I plan to use mlibc in the future.
proud nova
What do you mean by relocation here? Relocation as in ELF linking?
lost schooner
I have to go to school now, so Ill have to test without SMP later.
lost schooner
What do you mean by relocation here? Relocation as in ELF linking?
Relocatable object files. Things like shared libraries or position independent executables.
proud nova
Interesting
I have relocation working, but my userspace isn't finished (needs more syscalls) and I don't have libc yet
lost schooner
with SMP off, the outcome varies. The first time I attempted to boot, I saw some messages about uACPI init, but then it triple faulted half way through a kernel panic. The second time, it just froze before anything was printed.
A triple fault on my test device is implemented as a freeze
I'm suspecting that there is some kind of memory corruption.
Mostly likely uninitialised memory
dusky orbit
thanks <@337185227719966721> for your help
no problem, glad you figured it out 🙂
lost schooner
For some reason QEMU lets me allocate 4 cpus and 8GiB of RAM, when that is all the resources of my host machine
CPU 1 starts successfully, CPU 2 triple faults before it even reaches long mode
CPU 3 doesn't even get a chance to start
lost schooner
even with KVM on?
uneven minnow
Doesn't it just spawn host threads that it runs in
hollow trail
You can spawn any amount
lost schooner
CPU 1 starts successfully, CPU 2 triple faults before it even reaches long mode
this ^^^ happens when it tries to jump to long mode
uneven minnow
this ^^^ happens when it tries to jump to long mode
What exceptions do you get before the tripple fault?
lost schooner
Its a page fault. Its because of the way I map the trampoline. I'm currently implementing a solution.
uneven minnow
Ah yeah was about to ask if your page table was correct
dusky orbit
I'm sure you've considered this but there is the limine smp feature, means a little less "used once" code in the kernel.
lost schooner
I'm sure you've considered this but there is the limine smp feature, means a lit...
yeah, but I want to be as bootloader independent as possible. Also, I want to be able to do SMP startup properly.
Now I'm back to the timer bug
its happening again for some reason
this time the I/O APIC registers are correct
dusky orbit
Are you abstracting the abstraction that is the limine protocol? 
What is the current bug?
lost schooner
the HPET timer not giving me any interrupts
there is some inconsistency. Sometimes it works, other times it doesn't
this looks fine:
I can confirm that it has nothing to do with the higher than usual amount of RAM. Its something to do with the 4 cores
dusky orbit
that looks fine, you've tested other devices with the ioapic?
ah yeah, I remember you said ps2 stuff worked
hmm, you're not trying to use the hpet from multiple cores are you (without some sort of lock)?
lost schooner
hmm, you're not trying to use the hpet from multiple cores are you (without some...
err maybe
I use the HPET for synchronising the LAPIC timers
dusky orbit
just make sure you protect any shared state
lost schooner
but this is before that happens
dusky orbit
I see
you unmasked that particular comparator in the hpet?
I think they have per-timer interrupt masks
also for calibrating other times you can poll the main hpet counter, I find it more striaghtforward than using interrupts (if thats what you're doing)
lost schooner
also for calibrating other times you can poll the main hpet counter, I find it m...
thats what I do
interrupts mess with everything when you need precise timings
The problem I'm experiencing doesn't occur every single time
only sometimes
dusky orbit
ah thats fun 🤔
lost schooner
As I asked here: #x86 message, I thought it might be because the interrupt occurs right when interrupts are disabled
dusky orbit
ah I opened that and forgot to reply lol
the local apic queues interrupts (only 1 instance per vector) and will dispatch them when the cpu is ready to receive (IF is set)
lost schooner
iirc I don't think the i8259 PICs had that
lost schooner
ill have a look at your code again in a bit
I'll commit the latest code quickly
dusky orbit
no rush, Im being summoned to dinner
lost schooner
haha. What timezone are you in?
dusky orbit
aest/aedt I forget which it is, I live in sydney australia 🙂
lost schooner
aest/aedt I forget which it is, I live in sydney australia 🙂
oh nice. Syndey is AEDT. I'm up in Brisbane, which is AEST.
dusky orbit
oh nice 😄
I like brisbane, I find it a very calm city. I was actually up there over xmas/ny.
lost schooner
The problem I'm experiencing doesn't occur every single time
back to this ^^^, I can confirm that PS/2 keyboard interrupts still work. This means it must be a HPET-specific issue.
the issue only seems to happen when ubsan is on, but I'm not 100% certain as it doesn't happen every time when ubsan is on.
I did see a series of page faults once, but for some reason I thought it would be a good idea to kill QEMU instead of pausing it
I just got them again
it looks like some weird is happening when trying to write to the framebuffer on CPU 1. Then a large number of page faults follow as the stack frame before trying to write to the framebuffer is bogus
I can confirm that the HPET issue does happen very rarely when ubsan is off
the PS/2 keyboard interrupts are still fired though
the HPET issue can't be a race condition as this is before the other CPUs are actually allowed to do anything. They are online, but they complete a very basic init and wait for the scheduler to allow them to initialise their timers and run something.
slightly concerning:
also, for some reason my stop IPIs aren't working anymore
lost schooner
slightly concerning:
the address in this ^^^ is the exact address after the jump from SMP trampoline into the main kernel code.
which is why this is concerning
that should be unreachable
also how is that possible
that error is after the Scheduler is running, so that CPU should be running something given to it by the scheduler
the only way I can see it getting to that address is if it returns from __attribute__((noreturn)) functions.
multiple of them
thats not even whats happening. It looks like the scheduler is stuffing up somehow
I dont know how or why
I somehow got GDB to freeze whilst trying to work out what is happening
its not even responding to CTRL+C
hardy lake
also, for some reason my stop IPIs aren't working anymore
Probably because IPIs are maskable like any other irq
lost schooner
Probably because IPIs are maskable like any other irq
Its an NMI though
hardy lake
then it's spooky actions from a distance
lost schooner
This isn't good:
I was just trying to get the PIT working again
Its a stupid thing
I forgot to add a check before calling the function. That error was occurring very early in boot before ACPI tables are parsed
PIT is working again
I just learnt that once the LAPIC is enabled, its interrupts must go through the I/O APIC. The i8259 PICs only work for PS/2 interrupts and nothing else once the LAPIC is enabled.
this ^^^ occurred because I was assuming the i8259 PICs still work properly once the LAPIC is enabled.
lost schooner
also, for some reason my stop IPIs aren't working anymore
Got this ^^^ fixed. I hadn't updated my code for issuing it from a kernel panic after I changed my IPI system.
lost schooner
I've successfully converted everything over to using uacpi_table_find_by_signature instead of my old ACPI table finding code.
lost schooner
PIT is working again
I've decided I won't use it though. Its easier and also a better option to just continue using the HPET.
lost schooner
I just got a GPF in x86_64_LocalAPIC::InitTimer() because the address of the x86_64_LocalAPIC class is some weird value.
I also discovered that for some unknown reason there is 2 nodes in the schedulers list of processors for that processor
which is really strange
that still doesn't explain how spinlock_acquire gets called with an address of 806be00000000010
how is that possible
the address of the x86_64_LocalAPIC class is fine
the GS base is fine
Kernel GS base that is
no task switches have occurred on that CPU as this is before the scheduler starts
I have no clue how this could even happen
dusky orbit
I just learnt that once the LAPIC is enabled, its interrupts **must** go through...
ah yeah - the PICs and apic system dont work together. Usually the apics can be set up to be transparent for the PICs, but once you start using them directly thats not reliable.
that includes the ps2 controller
dusky orbit
I have no clue how this could even happen
possible memory corruption?
would be annoying, as that sucks to debug
lost schooner
Everything looks fine. I have no idea how it even happened. I still have QEMU and GDB open because I don't want to close it and risk not catching it for a long time.
dusky orbit
haha yeah I know that feeling
dusky orbit
Kernel GS base that is
when you say this, is that because you're reading the kernel_gs_base msr directly to get core-local info?
just checking you havent made the classic blunder of thinking that kernel gs base is what you get from reading at gs:0 while in the kernel.
lost schooner
No I read the MSR directly when possible.
The only exception is in my syscall handler where I use a combination of gs:<offset> and swapgs.
lost schooner
ah yeah - the PICs and apic system dont work together. Usually the apics can be ...
It was really strange. I don't mix them at all. When I did mix them was just in between implementing support for the LAPIC and I/O APIC. The PICs are always disabled and spurious interrupts that somehow go through are ignored. I exclusively use the APICs.
lost schooner
possible memory corruption?
I really hope it isn't. I hate memory corruption. Hopefully its just a small oversight somewhere.
lost schooner
I tried clearing the CD and NW bits in CR0 when attempting to enter long mode, but the extra instruction is just enough to go over the threshold of where the long mode should start at.
all fixed now
I should probably have mentioned that I am talking about my AP trampoline
I can't believe one tiny and instruction is enough to put the protected mode section of code over its allocated section
lost schooner
time for another real hardware test
The MADT table isnt found somehow
That shouldnt even be possible. It has worked before
This is the code for finding the table:
uacpi_table* MADT;
uacpi_status rc = uacpi_table_find_by_signature("APIC", &MADT);
if (rc != UACPI_STATUS_OK) {
PANIC("MADT table not found");
}
uACPIs error message is "not found"
@hollow trail do you have any idea why this could be happening?
hollow trail
memory corruption maybe? or this thing called too early
lost schooner
memory corruption maybe? or this thing called too early
wdym too early? that code works for QEMU
as far memory corruption, maybe I should look into implementing KASAN
hollow trail
add printfs to uacpi maybe
see what the internal structures contain, what table it iterates
tables.c file
lost schooner
Here is the result if I print the table signature in the do_search_tables function:
No APIC in that list
hollow trail
@lost schooner can you print the signatures of the tables being appended in uacpi_table_append
lost schooner
<@983999369848193044> can you print the signatures of the tables being appended ...
This is from the end of that function:
Thats the same list as in the image above
lost schooner
I don't know what kind of relevance this has, but when I run it on VirtualBox, it triple faults not long after entering ACPI mode.
This is probably some other bug though
hollow trail
virtualbox is known to work fine so thats definitely a you bug
hollow trail
This is from the end of that function:
okay so
i would say try managarm
but before we do that
this is where the XSDT gets processed
maybe add some logging/tracing here
maybe there's a bug or something you're mapping incorrectly
lost schooner
Alright. Its my mapping function
I just ran it with a proper mapping being replaced with just adding the hhdm offset
And it works
In a way
hollow trail
Thats a classic
lost schooner
My TTY scrolling is messed up
Which isnt surprising
Ill try video it. I found it quite interesting
hollow trail
Alright. Its my mapping function
Maybe you were returning an aligned pointer disregarding the offset within page or something
dusky orbit
thats fun to watch
lost schooner
my scroll text function is assuming that the bpp is 4, and width * bpp = pitch, neither of which are true on my test device
lost schooner
Maybe you were returning an aligned pointer disregarding the offset within page ...
That was it
Now it gets stuck much later. Still not sure exactly where
lost schooner
I just caught the rare chance where the HPET doesn't give an interrupt
no idea why it is happening
All the I/O APIC registers are correct
PS/2 interrupts work
I did have to manually call the PS/2 initialisation functions from GDB as the freeze due to the HPET is before normal PS/2 initialisation
The HPET GeneralINTStatus register is 0
the GeneralCAPID and GeneralConfig is correct
Interrupts are enabled though
the TimerConfigCAP is correct for the relevant timer
The only possible reason I can think of is that some annoying interrupt occurs at a poor time when starting the timer which causes the timer comparator value to be incorrect
the simple solution of cause is to disable interrupts during timer starting
dusky orbit
how soon are you setting the timer to fire?
if its near enough that could be case, or your code taking long enough that the expiry time has already passed by the time its written
lost schooner
hollow trail
Bruh
dusky orbit
yeah that would help ensuring the interrupt gets delivered
generally you want arming/starting a timer to be the last thing you do for its setup
lost schooner
hopefully that was the issue
but its hard to know for certain
time for more real hardware testing
It freezes (most likely triple faults) during namespace initialisation
Its probably my fault though
hollow trail
yeah 99.9%
lost schooner
Prior to the freeze, a bunch of PCIe reads/writes occur
I only support doing stuff with the root bus, so there is a chance it is trying to access another bus, and things dont go well.
I'll try running uACPI with UACPI_LOG_DEBUG
that might help
lost schooner
either way running with debug logging should hopefully help
its going to take ages though
hollow trail
yes
lost schooner
Its because I dont know how to configure other buses and interact with them
Anyway scrolling is even slower on real hardware than in QEMU
It feels like this is going to take half an hour at this rate
lost schooner
No
haughty galleon
understandable
dusky orbit
lol
lost schooner
I use my own TTY and framebuffer code.
And how would flanterm be any faster
Anyway, its been executing the same AML function for ages
hollow trail
lol
hollow trail
Anyway, its been executing the same AML function for ages
function or namespace load?
lost schooner
Namespace init
Namespace load didnt take long I dont think
I could be wrong though
haughty galleon
And how would flanterm be any faster
you'd be surprised
lost schooner
if its relatively modern hw
Its 2018 or 2019 I think. The BIOS is very recent though.
lost schooner
you'd be surprised
What does it do to make scrolling faster?
scarlet hollow
lost schooner
35 minutes after it started running AML, its still going
hollow trail
Yeah im afraid you fucked it up somehow
Thats not possible
And if its an infinite while that would get aborted after 30 seconds
Unless it logs like one line per hour
Is your allocator O(N)?
lost schooner
It might be in namespace load. Its been an hour, and its still going
lost schooner
Is your allocator O(N)?
My heap is
My VMM is O(nlog(n))
The scrolling is really slow. Its like 2 lines a second
Maybe 3 or 4
Is there a way I can run QEMU with ACPI tables dumped with acpidump?
haughty galleon
What does it do to make scrolling faster?
well, try it lol
lost schooner
Finally, its actually doing something. I just saw a total of 8 reads from either normal IO or PCIe
Its been 70 minutes since I started it
haughty galleon
have you ever tried running the lai test image?
lost schooner
Why LAI
hollow trail
Finally, its actually doing something. I just saw a total of 8 reads from either...
So did it hang?
lost schooner
Its still going
lost schooner
Its the scrolling thats making it slow
How messed up is the AML for it to take this long
lost schooner
But it will write to non existent mmio
So, would I just not do the MMIO?
hollow trail
Maybe
haughty galleon
Why LAI
lots of scrolling text, uses flanterm
other than that, uACPI is superior to LAI in every way lol
lost schooner
I think I'll try running QEMU with the ACPI tables from my test device. Its been running on my test device for 1 hour and 45 minutes now, I'm just going to stop it. It doesn't look like it will finish any time soon.
lost schooner
-acpitable path=...
that doesn't work:
qemu-system-x86_64: -acpitable res.txt: ACPI table too big, requested: 562438, max: 65535
tired cedar
lost schooner
still doesn't work:
qemu-system-x86_64: -acpitable dsdt.dat: ACPI table too big, requested: 73830, max: 65535
yeah nvm
u cant
nya guy also ran into this
i mean if mint says flanterm has faster scrolling try that
are you not using pat?
lost schooner
No
It wont take long to set it up though
Anyway, I have to go, so I'll have to sort that out tomorrow
haughty galleon
Limine default maps the framebuffer to WC with the PAT
but in any case Flanterm is very fast even without PAT
lost schooner
I remap everything for obvious reasons. I might try flanterm if the PAT doesnt help.
hollow trail
Limine default maps the framebuffer to WC with the PAT
i thought it changed MTRRs
haughty galleon
I remap everything for obvious reasons. I might try flanterm if the PAT doesnt h...
the PAT is already defined in the Limine spec
which means that really you only have to change the page tables flags
unless you have edited the PAT yourself
hollow trail
it did... like 3 years ago
seems like a more reliable way (or at least more useful)
although mtrrs are pretty limited ofc
and idk if limine does any range collision detection for mtrrs
haughty galleon
and idk if limine does any range collision detection for mtrrs
yes but also no, it's not more reliable
a lot of machines simply do not have enough variable range MTRRs so you cannot do it on those machines
hollow trail
thats true
but you can say in the spec that you do both
and that mtrr configuration is only done if possible
haughty galleon
nah, sounds too complicated, and plus setting up the PAT and 2 bits in PTEs isn't really complicated
hollow trail
how is it too complicated?
haughty galleon
it complicates the spec by adding extra, mostly useless, redundant features
it's something i try to avoid
hollow trail
didnt u want to add flanterm configuration from limine spec
haughty galleon
didnt u want to add flanterm configuration from limine spec
well i did decide against that, didn't i though?
plus, that technically wouldn't be redundant
lost schooner
I've been having a break for a few days, but now I'm back. I am currently updating to GCC 14.1.1
lost schooner
also updating to binutils 2.42
lost schooner
The updated versions appear to work well
lost schooner
I guess its time I implement PAT support
hardy lake
I guess its time I implement PAT support
It's not that complicated, you just set bit in pte
then poof it works
idk if you have to flush the tlb
because of the different caching setting
Probably
btw you only set the bit in the last level of the PTE
not the higher levels
iirc
lost schooner
I know
I assume it works in the same sort of way as NX, where you only want to set it in the lowest level
lost schooner
I just quickly updated to C++23 and C23 instead of C++20 and C17 now that I am using GCC 14
hardy lake
I assume it works in the same sort of way as NX, where you only want to set it i...
uhh what
it works the same as any other pte bit
it must be set in its parents' tables for it to apply
lost schooner
huh
hardy lake
unlike the caching bits, which iirc according to the sdm are only looked at in the last level
lost schooner
I know for Write and User, it must be set in the parent tables, but for NX you only set it in the lowest level unless you want all the sublevels to be non-executable.
hardy lake
which is the same for Write and User
lost schooner
And I was saying that I assume the caching bits work in the same way as the NX bit, where you only want to set it in the lowest level
hardy lake
well the caching bits only matter for the last level
iirc the processor ignores them otherwise
unlike XD
lost schooner
Anyway, now that I have (hopefully) finished getting sidetracked, I will finally implement PAT support
lost schooner
I have successfully implemented PAT support, now I just have to make use of it.
lost schooner
tried mapping the framebuffer as WC, but ended up getting a reserved-write page fault...
lost schooner
this makes no sense
at no page level is there a reserved bit set
actually, nevermind
one is set at PML4 for some reason
I have no clue why
I thought I might have known why, but obviously not
I thought it was because I was forgetting to mask out the PAT bit in the higher page table levels, but that bit is still set
I'm stupid
I was masking out the wrong bit
lost schooner
all of this doesn't seem to have changed much in QEMU, but hopefully it improves scrolling performance on real hardware
lost schooner
time to test this on real hardware
I was just doing a quick test in QEMU before I copied it to a USB, and I got a triple fault
during startup on CPU 1
CR2, RAX, RDI, RCX, and RSP are very odd
dusky orbit
yeah thats very suspicious, so is RIP
unless your ap bootstrap lives in the nullpage
hollow trail
Bruh
lost schooner
unless your ap bootstrap lives in the nullpage
yes it does
RIP is fine
After a bit of messing around I have managed to get a raw memory dump of the first page in a binary format (not QEMU's annoying ELF format)
Now I think about it, RAX, RDI, RCX, RSP are full of repeating 0x90, which is the opcode for nop
most likely meaning that something is getting read before it should...
I'm confused
the start lock is 1, meaning that the AP is not allowed to read most values
yet it still somehow manages to
how does this not work:
.spin:
mov eax, dword [0xFF8] ; start lock
test eax, eax
jnz .spin
the DWORD at 0xFF8 is 1
dusky orbit
I havent done locks in x86 asm, but that looks too easy
lost schooner
its the simplest way I could think of doing it, and I've never had any issues with it until running it just now
its so strange
hardy lake
maybe try lock cmpxchg
celest sierra
bts
lost schooner
Hopefully that issue was just a one-off. If I see it again, I'll have to make some changes
For some reason I feel like implementing KASAN
I have no idea how though...
lost schooner
Just reading through the algorithm stuff, and its quite interesting. I have no idea how to implement shadow memory though
I saw something in the intel sdm about a shadow stack, but idk how that is related
lost schooner
This is an interesting kernel panic:
Interrupt occurred. RIP: 0000000000000092 Interrupt number: 80839f78
UBSan: out of bounds at /home/frosty515/dev/WorldOS/WorldOS/kernel/src/arch/x86_64/interrupts/isr.cpp:141:68
KERNEL PANIC!
UBSan: out of bounds at /home/frosty515/dev/WorldOS/WorldOS/kernel/src/arch/x86_64/interrupts/isr.cpp:141:68
on CPU 0 (LAPIC 0)
RAX=000000000000006d RBX=ffffffff808398b8 RCX=ffffffff8083985c RDX=00000000000000e9
RSI=000000000000000a RDI=ffffffff808398b8 RSP=ffffffff80839858 RBP=ffffffff80839880
R8 =0000000000000003 R9 =0000000000000000 R10=0000000000000003 R11=0000000000000000
R12=ffffffff80839f78 R13=0000000000000000 R14=ffffffff80839d98 R15=0000000000000070
RIP=ffffffff80079dc0 RFL=0000000000000086
CS=0008 DS=0010
Stack trace:
ffffffff80079dc0: sanitiser_panic
ubsan violation whilst handling an unusual interrupt
that looks way too much like stack corruption
an interrupt number of 0x80839f78?
thats odd
definitely stack corruption
the ubsan violation is because of the very large interrupt number
and that RSP looks unusually high
I just discovered that the freeze on real hardware is immediately after a uacpi_kernel_sleep(1) call
Also, the WC framebuffer mapping had no noticeable effect on scrolling speed.
lost schooner
It looks like the HPET timer isn't issuing interrupts at all on real hardware
dusky orbit
I saw something in the intel sdm about a shadow stack, but idk how that is relat...
shadow stack isnt related, thats just a copy of the call stack somewhere else in memory
nice feature for security purposes though
and mangarm has an asan implemenation you can look at it, its pretty straight forward. Lots of it is logging stuff.
uneven minnow
nice feature for security purposes though
But what if the shadow stack is also broken O_O
lost schooner
I was looking at @hardy lake's implementation earlier, and it seems fairly straight forward
dusky orbit
But what if the shadow stack is also broken O_O
you have to find it first 😛
uneven minnow
you have to find it first 😛
It's by the lake right next to the city center, siting on the first bench in the park near the starbucks :>
lost schooner
After seeing the stack corruption earlier, I was also thinking that I should add guard pages either side of the stack to protect against overflows
and as another protection measure, using ISTs could be helpful
dusky orbit
guard pages are nice, would recommend
lost schooner
turns out that my VMM had most of the needed stuff to create guard pages effectively. All I need to do is reserve a region with 2 pages of extra space than the stack normally needs, then map the region a page into, and a page short from the end of the previously reserved region.
lost schooner
I didn't realise it's been a week since I last gave an update
I'd been having a break from osdev, but I'm back now
Whilst implementing guard pages on my kernel stack, I've discovered many things that I forgot to change after implementing SMP. Some of these include, but are not limited to:
- I forgot to not use the early boot stack when a kernel task ends, which meant that if multiple kernel tasks tried to return at once, their stacks would overrun each others.
- I was switching to the wrong stack when trying to end the current user thread, which would have a similar outcome as the previous bug.
- The BSP would always get a bigger stack for interrupts as it would still use the early boot stack.
now I think I'm dealing with a triple fault, but I'm not certain...
QEMU froze (I don't know if it had paused automatically or not), so I attached GDB, and the image below shows the unique stack trace I got. CPU 1 was online doing something in the SMP trampoline (I don't know what just yet), CPU 2 and CPU 3 are not online yet. I tried unpausing from GDB not realising that QEMU was most likely already paused. I don't know what happened as I didn't give QEMU -d int.
actually, I wonder if its CPU 1 that triple faulted....
yep it is
turns out I was never initialising the final kernel stack in the AP trampoline correctly, and with the guard pages, it instantly page faults on a push.
now it works
time for a real hardware test as I'm bored
the HPET isn't working
hollow trail
It freezes (most likely triple faults) during namespace initialisation
Did you figure it out?
lost schooner
As I said just above, its the HPET not working
lost schooner
alright, the HPET is working
what was happening is really weird
the HPET works, but wouldn't send interrupts when they are needed for uacpi_kernel_sleep. They work perfectly fine after uACPI is fully initialised.
I ended up just implementing uacpi_kernel_sleep in a very similar way to uacpi_kernel_stall as that's essentially what kernel mode sleeping does anyway.
For the first time, uACPI fully initialised in WorldOS on real hardware
celest sierra
so you just spin until the deadline?
lost schooner
yes
lost schooner
I'm fully aware
celest sierra
sleep is usually implemented in terms of setting up a timer and blocking the current thread on said timer
lost schooner
but at this stage in the boot process, the scheduler isn't initialised, so threads don't exist.
lost schooner
sleep is usually implemented in terms of setting up a timer and blocking the cur...
I know. Thats what I do for userspace sleep
celest sierra
in that case you should set up a kernel task/thread as soon as possible
so you don't have to hack around this shortcoming
by implementing sleep as a spin loop, for example
lost schooner
I do have kernel tasks and threads, and they can sleep, but I haven't fully implemented how to trigger that. Also, when I initialise uACPI, the scheduler isn't running.
lost schooner
the HPET works, but wouldn't send interrupts when they are needed for `uacpi_ker...
I just realised why this ^^^ weirdness happens. I do ACPI full initialisation before fully initialising the kernel clock.
I'm stupid
Yep, that was it
Now I am dealing with the PS/2 controller being difficult
Also why does my laptop have 12 AML blobs
celest sierra
sounds about right
lost schooner
In the past I have noticed that my PS/2 driver doesn't work on virtual box, but couldn't be bothered to work out why
hollow trail
For the first time, uACPI fully initialised in WorldOS on real hardware
nice
hollow trail
Also why does my laptop have 12 AML blobs
yeah thats normal
hollow trail
In the past I have noticed that my PS/2 driver doesn't work on virtual box, but ...
virtual box is the only vm that simulates hw delays
thats why it doesnt work for you
lost schooner
virtual box is the only vm that simulates hw delays
that would explain why the reset times out
I probably just need to increase the amount of attempts
as for my test device, I have no idea why the self test would fail
hollow trail
probably because u read some other byte and not the result of the self test
i had to make like military grade byte discard for my old driver
that ps2 driver was insane
but it worked everywhere
lost schooner
so a fail should strictly be a 0xFC response:
anything other than those 2 options should be ignored?
hollow trail
yeah
hollow trail
so a fail should strictly be a 0xFC response:
lost schooner
why does that seem so complex
hollow trail
because ps/2 is racey and kinda shit
celest sierra
it's not that bad, it's just really handling a lot of edge cases
hollow trail
lost schooner
would this be enough:
SendCommand(PS2_CMD_SELF_TEST);
unsigned char selfTestResult;
while (true) {
selfTestResult = ReadData();
if (selfTestResult == 0x55 || selfTestResult == 0xFC)
break;
}
if (selfTestResult != 0x55) {
PANIC("PS/2 controller self test failed!");
}
it also supports the multiplexing ps/2
e.g. thinkpads with 2 mice
hollow trail
yeah and it has a touchpad driver too
hollow trail
yeah it was probably inspired by it
celest sierra
hollow trail
this code is 3 years old
celest sierra
those verbose names and lots of interesting types
and helper functions
i like how it all reads
hollow trail
lol yeah, at least coming back to it after 3 years its still kinda readable
lost schooner
except @hollow trail's code is probably better than some of the serenity stuff
celest sierra
definitely, serenity kernel is terrible
its just a bunch of fancy code but its all pretty bad considering how much work was put into it
hollow trail
yeah i always had a thing for driver stability so i would bang my head against the wall until it worked on absolutely every computer i own
hollow trail
its just a bunch of fancy code but its all pretty bad considering how much work ...
yeah its all surprisingly bad
celest sierra
its capable and its got some cool stuff, but i feel like its one blow away from falling apart lmao
hollow trail
they have mutexes being taken with irqs disabled all over the place
so it falls apart rather quickly
celest sierra
its got unix permissions, lots of drivers, a decent netstack (afaik)
but yeah
and the big process lock™️
hollow trail
*lots of drivers where people started them because it seemed fun but none that are finished or good
lost schooner
and the big process lock™️
that thing is horrible
they are trying to get rid of it, but it should never have existed in the first place
hollow trail
they have mutexes being taken with irqs disabled all over the place
and it still falls apart with smp almost instantly
even with a big process lock
celest sierra
basically a shitty solution, by mr kling himself, to a problem that should have been handled properly
and yeah, not even a good solution apparently
its so sad they get so much attention
hollow trail
they started getting rid of it for select syscalls a few years ago
lost schooner
would this be enough:
```cpp
SendCommand(PS2_CMD_SELF_TEST);
unsigned char selfT...
this ^^^ worked. Now I just have to do something similar for the individual ports.
hollow trail
but since the os was abandoned since the last year that basically got nowehere
hollow trail
would this be enough:
```cpp
SendCommand(PS2_CMD_SELF_TEST);
unsigned char selfT...
yeah
lost schooner
they started getting rid of it for select syscalls a few years ago
its almost entirely gone now, but no one is making an effort to improve it with all the web browser stuff
hollow trail
would be nice to have it bounded by some amount of retries but whatever
lost schooner
would be nice to have it bounded by some amount of retries but whatever
yeah, I'll do that. How many retries do you think would be good?
hollow trail
3 or 5
celest sierra
i hope it gets some action going again now that they separated ladybird from serenity
its fun seeing the progress regardless of how shit their kernel is
hollow trail
yeah i wonder what will happen to it
but im afraid nothing good 
most core contributors moved to the browser
lost schooner
most core contributors moved to the browser
including mr kling
lost schooner
e.g. thinkpads with 2 mice
why??? Wouldn't that mean 3 PS/2 devices assuming that there is also a keyboard? PS/2 controllers are only dual channel iirc
celest sierra
they are multiplexed? i think
hollow trail
multiplexing ps/2 supports up to 6 devices
celest sierra
damn
hollow trail
and yes its 2 mice and a keyboard
lost schooner
multiplexing ps/2 supports up to 6 devices
oh. How would that work?
celest sierra
is the 2 mice the nipple + touchpad?
hollow trail
lost schooner
is the 2 mice the nipple + touchpad?
probably
hollow trail
is the 2 mice the nipple + touchpad?
yup
celest sierra
nice, fun stuff
hollow trail
with the ability to identify touchpad with a magic sequence
it has special ps/2 packets and stuff
also very cool
lost schooner
I'll have a read of that later
Doesn't mean I'll implement support for it though...
hollow trail
i did it because i couldn't scroll my terminal output from real hw
so i implemented a touchpad driver with gesture support
it tells you the x and y of every finger
and the pressure they apply on the touchpad
you can have a state machine for advanced gestures
hollow trail
i did that too but that was kind of annoying cause i had to reboot into linux
celest sierra
implement multi touch support and make the terminal scroll ✅
hollow trail
the ahci driver was also bulletproof
celest sierra
i expect nothing less of you at this point :^)
hollow trail
lol
celest sierra
cant wait for ultraOS return
hollow trail
yeah same
but im gonna be stuck with the basics like you
making a good allocator and stuff
ultraos had cool stable drivers and was smp stable but used absolutely shit algorithms
celest sierra
yea i think im just gonna use frigg instead of reimplementing half of the stl
hollow trail
like it was a bitmap pmm
hollow trail
yea i think im just gonna use frigg instead of reimplementing half of the stl
good call
i did implement an rb tree myself for it
and i hated every minute of it
hollow trail
i dont even want to know how an rbtree works <:tf:968899548258914394>
neither do I tbh
im terrible at formal stuff like this
lost schooner
Also, the WC framebuffer mapping had no noticeable effect on scrolling speed.
this ^^^ is actually wrong. It has a massive impact, its just that the framebuffer reading is really slow. I know this because writing to the screen is very fast, but scrolling is still very slow.
celest sierra
you should not be reading from the framebuffer
lost schooner
I did setup double buffering at some point to solve that, but I never did it properly, so its very broken.
Most of the time I just have it disabled
celest sierra
you should take a look at how flanterm handles redrawing the terminal
it's very efficient in terms of redraws
hollow trail
Reading from it is very very slow
It cant be optimized with wc
Its an entire pci transaction
lost schooner
Now I'm back to the controller self check failing
There are 5 attempts for it to send a valid value and it fails. When I had no attempt limit, it was successful
lost schooner
its over I/O ports though
actually, I see a potential issue
I don't actually check if there is data available in my x86_64_8042_ReadData function
hollow trail
bruh
lost schooner
and now when I try do that, it gets stuck on an in instruction whilst trying to read the status port
hollow trail
hows that possible even
lost schooner
idk
lost schooner
It was a stupid mistake. It wasn't actually stuck on that instruction, it was just constantly reading from the status port. That was because of the way I clear the output buffer before reconfiguring the controller.
Its fixed now
time to see if it works on real hardware
PS/2 initialisation completes successfully now
Although, the keyboard type is unknown
My laptop apparently has 126 devices according to uACPI
Anyway, now I have to deal with an annoying page fault
There is too much glare this time of day for me to take a photo of what is happening
the error is whilst trying to load the symbol table file from the initramfs
this is prior to proper initramfs loading
It just finds the offset within the initramfs, which is loaded in memory by limine
anyway, I have no idea why this would fail
it should be mapped
it also happens on QEMU when I set the RAM amount to 8G
dusky orbit
hmm, you're sure the offset is correct?
and should be mapped in one thing, but you've checked it is?
lost schooner
the address is in the first page that isn't mapped, which leads me to believe something strange is happening
it was more stupidness
dusky orbit
at least it was a quick fix
lost schooner
I had forgotten about kernel memory map entries in part of the memory mapping code
time for another real hardware test...
Well, it gets a lot further in boot before triple faulting
It successfully detects 27 PCIe devices
And the VFS root gets mounted.
Also, all of this means that the scheduler is running
I don't know if I have mentioned this before, but it is a massive pain to do real hardware tests as my only x86_64 device is my development and test device. So everytime I want to do a test, I have to reboot my laptop.
Anyway, here is a list of the PCIe devices (in order):
1 × 11:80:00
1 × 08:80:00
1 × 03:00:00
1 × 04:03:00
1 × 07:80:00
1 × 00:50:07
1 × 01:06:01
3 × 06:04:00
1 × 0C:03:30
12 × 11:80:00
1 × 06:01:00
1 × 0C:05:00
1 × 01:08:02
1 × 02:80:00
In Class:Subclass:ProgIF format
Most of those are intel devices except the last 2
hardy lake
if you want (and it doesn't nuke my hard drive or dox me)
you can send me isos
of worldos
and I can use my test subjects
for testing
it
at the end of thorough testing
lost schooner
Nah, I'm fine. I won't be doing constant real hardware testing for much longer. Only the occasional testing. I just need to work out why it triple faults (or maybe a freeze, it's hard to know due to triple faults being implemented as a freeze) in between loading the initramfs and mounting the VFS root.
dusky orbit
Nah, I'm fine. I won't be doing constant real hardware testing for much longer. ...
something I wish I'd added sooner was an uptime counter to any in-use framebuffers, so I can see if the kernel has locked up or a particular task has frozen.
also you dont get to handle triple faults 😛
lost schooner
Isn't there something I can do with ISTs to make triple faults less likely to occur?
dusky orbit
probably, but if you get a triple fault something has gone wrong
if you get a double fault (which you can handle), you are already in trouble
dusky orbit
Isn't there something I can do with ISTs to make triple faults less likely to oc...
I know this isnt super helpful advice, but that sounds like treating the symptoms not the cause
lost schooner
I'm suspected a triple fault due to something stack related, but I have no way to know for certain
lost schooner
if you get a double fault (which you can handle), you are already in trouble
I don't get a double fault that can be handled currently
if it is stack related, is there a way I can use some alternative stack?
dusky orbit
yeah you can use the ISTs, like you were thinking
but if you're not getting a double fault at all, it could also be an issue with your idt
like it gets unmapped (page tables getting trashed somehow?)
but a known-good stack would rule that out as the issue
lost schooner
if its a stack overflow, it would go into the guard page, causing a page fault. But since this is all in kernel mode, the exception handler would use the same stack, causing a instant double fault followed by a triple fault. I assume I would have an IST for PFs, GPFs, and double faults???
I'm not exactly sure how ISTs work so I'll have to have a read of the intel sdm
dusky orbit
if its a stack overflow, it would go into the guard page, causing a page fault. ...
yeah that could be the issue
dusky orbit
yeah theyre super simple
lost schooner
that will be quick to implement
dusky orbit
I have 😦
lost schooner
im just going to ignore it and if it happens again, I'll deal with it
lost schooner
well I get a ubsan violation, but that is interrupted by the LAPIC timer, resulting in this madness.
so maybe I should disable interrupts the moment a ubsan violation occurs
that might only be a temporary solution though. Something much worse could be happening
dusky orbit
yeah I agree, disabling interrupts for ubsan shouldnt be necessary
maybe thats a lead on the corruption?
lost schooner
well, my ISTs are working well. Just saw some stack weirdness occur, and it was able to be handled
for some reason, the stack is in a guard page
at entry to a function
this is happening on entry to the first thing the scheduler executes on that CPU
dusky orbit
when setting the stack of a new thead, you're using the highest address of the memory you allocate?
dusky orbit
well, my ISTs are working well. Just saw some stack weirdness occur, and it was ...
thats good, nice
lost schooner
when setting the stack of a new thead, you're using the highest address of the m...
I assume so?
i should probably verify that
dusky orbit
i would also assume so, but just checking
lost schooner
first of all, I had forgotten to redo my stack creation code in the scheduler after adding guard pages to the kernel stack
lost schooner
kernel stack under/over flow detection is working
lost schooner
hollow trail
Nice
lost schooner
actually, I think that is a genuine kernel stack overflow
idk how I managed that
yes it is
at entry to Kernel_Stage2
that looks suspiciously like the stack being given to that thread hasn't had the stack size added to the base address
how is that even possible
I use this code for getting the stack:
if (thread->GetParent()->GetPriority() == Priority::KERNEL && thread->GetKernelStack() != 0) {
dbgprintf("Kernel thread already has stack %lx\n", thread->GetKernelStack());
thread->SetStack(thread->GetKernelStack());
regs->RSP = thread->GetStack();
}
and the dbgprintf returns the correct stack
yet the RSP is towards the beginning of the allocated region
oh its so stupid
why is GCC allocating room for all stack variables in a function at entry
I always thought it was only when needed
that explains the problem though
real hardware testing time
PS2 device 2 identification failed
I wonder why
Probably similar deal as the other PS2 issues.
This time I got a freeze during PS2 initialisation
Something odd is definitely still happening
lost schooner
I've decided I want to rename WorldOS
I don't know what I will rename it to
The current name was a suggestion from a friend just as I was starting this project
I googled WorldOS out of curiosity just to see if the github would show up, and discovered that the name WorldOS is trademarked.
lost schooner
Whats the name going to be
I don't know yet
dusky orbit
I've decided I want to rename WorldOS
big decision, good luck with choosing a new name
lost schooner
my first thought was FrostyOS, but that name is already taken by an android ROM
hollow trail
Why is your last commit 3 weeks ago
hardy lake
Why is your last commit 3 weeks ago
Maybe check other branches
hollow trail
Maybe check other branches
Nope
lost schooner
Why is your last commit 3 weeks ago
Because most changes are local
I'll commit it all soon
I've just had lots of stuff going on IRL the last few weeks.
hollow trail
i see
lost schooner
this is odd:
[void LinkedList::SimpleLinkedList<T>::insert(const T*) [with T = Scheduling::Scheduler::ProcessorInfo]] WARN: object already exists. Not inserting.
KERNEL PANIC!
UBSan: type mismatch at /home/frosty515/dev/WorldOS/WorldOS/kernel/src/Scheduling/Scheduler.cpp:395:62
on CPU 1 (LAPIC 1)
RAX=0000000000000067 RBX=ffffffff801839e0 RCX=ffffffff80837aac RDX=00000000000000e9
RSI=000000000000000a RDI=ffffffff80837af8 RSP=ffffffff80837aa8 RBP=ffffffff80837ad0
R8 =0000000000000003 R9 =0000000000000000 R10=0000000000000003 R11=0000000000000000
R12=ffffffff80837af8 R13=ffffffff80113a8f R14=0000000000000297 R15=ffffffff80837ee0
RIP=ffffffff8007b040 RFL=0000000000000082
CS=0008 DS=0010
Stack trace:
ffffffff8007b040: sanitiser_panic
ffffffff8007b0c9: ubsan_print_error
ffffffff80087eed: Scheduling::Scheduler::AddProcessor(Processor*)
looks suspiciously like a race condition somewhere
this is the line (the ^ is the column the error occurred on):
x86_64_LocalAPIC* LAPIC = processor->GetLocalAPIC();
^
that error from the linked list is would be because the same ProcessorInfo struct is being inserted more than once
I saw this error once before ages ago, but just chose to ignore it
lost schooner
when I looked into this with GDB i discovered that a page fault was occurring in x86_64_walk_stack_frames, so I have spent the last 20 minutes trying to make it impossible for it to page fault.
it would then get stuck trying to issue an IPI because the lists for all the processors are locked somehow
probably me just handling StopThis IPIs slightly wrong
yep
fairly simple solution though
lost schooner
this is the line (the `^` is the column the error occurred on):
```cpp
x86_64_Lo...
Just noticed this doesn't display properly on phones, so the error occurred on the column before the parentheses.
lost schooner
I don't think I mentioned this, but the above panic occurred while I was in the process of doing some testing on my PS/2 driver as I've been trying to make it less error-prone.
lost schooner
that error from the linked list is would be because the same `ProcessorInfo` str...
I think I might have found out this ^^^ is happening. For some unknown reason getCount() is being called on the list of processors when it isn't locked. It is a very simple fix luckily.
another annoying page fault:
so something is stuffing up the vtable for the instance of the PS2Controller class
which is really weird
why is the vtable 0x0
ubsan tooling is making the disassembly so hard to understand
it still happens with ubsan off. At least now the disassembly is readable
this worked yesterday. All I have changed is make the functions const
side note: GCC likes to spit out a whole load of -Wmaybe-uninitialized and -Wpendantic related warnings for uACPI
many of which are ISO C++ forbids flexible array member and ISO C++ prohibits anonymous structs
hollow trail
many of which are `ISO C++ forbids flexible array member` and `ISO C++ prohibits...
Just let it cope and seethe
lost schooner
why is the vtable 0x0
if I create the object on the heap, that isn't an issue anymore. It was probably a bad a idea for it to be a global object anyway.
lost schooner
Just let it cope and seethe
what do you mean? I was just going to add some pragma magic around where I include the files to make it quiet most of the time.
hollow trail
if I create the object on the heap, that isn't an issue anymore. It was probably...
Do you call global constructors?
lost schooner
no
lost schooner
I guess they should be called, but I'll have to work out when is appropriate
hollow trail
At the very least do new (&object) Object
hollow trail
I guess they should be called, but I'll have to work out when is appropriate
Did you not look at the osdev page about how to use c++ in a kernel?
It generates a list of things you must call
lost schooner
It would have to be at least after heap initialisation
hollow trail
It would have to be at least after heap initialisation
Placement new works in a predefined buffer
lost schooner
I know that, I'm more talking about when to do that
hollow trail
It doesn't need the heap
lost schooner
I'll have to set that up
hollow trail
I know that, I'm more talking about when to do that
Well
As long as you dont allocate in global constructors
You can do it early
lost schooner
I'll do it after very early init (including kmalloc init) just to be safe.
lost schooner
I just implemented placement new/delete, all the __cxa_atexit, __cxa_finalize, and __cxa_pure_virtual stuff, and all the local static variable guard stuff.
lost schooner
How do I even call global constructors?
I know that I could use .ctors, but is there a better way?
hollow trail
No
lost schooner
I had just realised that by looking at your old kernel's code for it
Alright that should be implemented now. I added the kernel symbols for dtors as well in case at some point in the future I want to add support for them.
well, calling global constructors at that stage breaks many things
I guess they need to be called much earlier and I just have to make sure none do heap allocation
dusky orbit
I would heavily advise against doing anything inside of a ctor for any object thats going to have a global instance
its not worth the trouble imo
lost schooner
when I run the addr2line of the addresses of the global ctors, most are at the end of seemingly random files
what ever is happening, lots of stuff is breaking
suddenly becoming null
I think I was better off not calling global constructors and just working around it
if I call them really early, not even printf is usable
but if I call them later, so almost all the basics are accessible, everything breaks
I decided that I would call them really early, which seem to work until I got this kernel panic right as the ACPI shutdown was almost complete:
KERNEL PANIC!
Page fault in kernel-mode at 0 while trying to execute a non-present page at address 0 on CPU 3 (LAPIC 3)
RAX=ffffffff802ce278 RBX=595e9fbd94fda766 RCX=00000000c0000102 RDX=0000000000000010
RSI=ffffffff8074cf30 RDI=ffffffff802ce240 RSP=ffffffff8074cf00 RBP=ffffffff8074cf10
R8 =0000000000000000 R9 =0000000000000000 R10=0000000000000000 R11=0000000000000000
R12=ffffffff802a3060 R13=0000000000000000 R14=0000000000000000 R15=ffffffff8074cf30
RIP=0000000000000000 RFL=0000000000000082
CS=0008 DS=0010
Stack trace:
0000000000000000
ffffffff80000b7a: isr_common
ffffffff80006b67: x86_64_idle_loop.l
its stack corruption I think
I probably should be stopping the other CPUs prior to ACPI shutdown anyway
hardy lake
I probably should be stopping the other CPUs prior to ACPI shutdown anyway
Ya think?
full needle
why is the vtable 0x0
show linker script?
at a guess missing .rodata.* and/or .text.*
full needle
ah
hollow trail
i hate it when the global constructors are not
are not? are not what?
muted blade
_ _
hallow bluff
are not? are not what?
when they are not
haughty galleon
what
lost schooner
I just did a quick test on virtual box out of curiosity, and it triple faults during uACPI init
which is really bad
it gets further on real hardware than in virtual box
but virtual box kinda sucks
and vmware is being difficult (as always) so I can't test on that
lost schooner
I've decided I want to rename WorldOS
almost a week has gone by, but I still haven't came up with a name
hollow trail
FrozenOS
lost schooner
Thats a good suggestion
Except that it already exists
Github organisation and everything
celest sierra
frosty os
lost schooner
So many annoying cereal ads
Doesn't look like there is a github org for it
I might try and make an org to confirm
The org exists but is empty
FrostyOS-dev isnt though
Looks like Im doing a rename
celest sierra
the name deserves a better owner
idk why those people that remix aosp remixes names their stuff anything with OS at the end
they arent making an operating system they just repackage aosp with custom apps 😭
lost schooner
First stage of the rename is done
Now I have to rename the toolchain and stuff in the build system and main code
FrostyOS (formerly WorldOS)
lost schooner
alright all the build system and main code stuff is changed
I'll deal with the toolchain tomorrow
i really don't feel like building GCC right now
lost schooner
time to fix my binutils and GCC builds
its currently 3:42pm. I'm going to guess that it will take at least 1 hour and a half to do all this
lost schooner
changes should be done now, just got to actually build it
lost schooner
toolchain is done. binutils took about 5 minutes, and GCC about 45 minutes
I also removed all the old makefiles
dusky orbit
Nice
Reminds me I should tidy my build system soon, it's grown unchecked for a bit too long
lost schooner
time to get back to whatever it was that I was working on before this
i think I was doing something related to stopping CPUs
I think I remember it not working, but when I just tested it, it is working
oh well
I think ages ago I was going to do detection to see if stuff like the 8259 pics, 8254 PIT, and RTC actually exist
i think I'm supposed to use uacpi_find_device maybe
ah here we go. it was all this stuff: #x86 message
lost schooner
I might even try use uacpi_get_current_resources to find the IO ranges
that won't work for RTC though as it uses CMOS registers
and looking at uacpi/resources.h it doesn't look there is a type for CMOS registers
lost schooner
alright, RTC detection is working
QEMU is running sooo slow
I can see the characters being printed to debug individually
since I'm bored, I'm going to see if this and the double buffering fix works on real hardware
The double buffering has a massive speed improvement
But, what on earth happened here:
lost schooner
its consistent
ohh
I think I know why
I don't allocate the buffer as the correct size, which probably stuffs many things up
Yep, that was it
Scrolling is so fast now
Now it freezes part way through printing info about the PS/2 controller
It initialises correctly though
lost schooner
Scrolling is so fast now
Its actually much faster (like 10x) on real hardware than QEMU, when previously, it was much slower (like 3x).
hollow trail
that won't work for RTC though as it uses CMOS registers
They're just io tho?
lost schooner
Yes, but its like 2 IO ports for a whole bunch of CMOS registers with different purposes.
hollow trail
Yes
Acpi is not supposed to describe the purpose
Just the system resources used
Which are two io ports in this case
lost schooner
I guess I could use it to determine what IO ports I should be using for doing CMOS stuff.
hollow trail
yeah
lost schooner
Now it freezes part way through printing info about the PS/2 controller
This isn't entirely correct. The freeze is whilst configuring the second PS/2 device
lost schooner
for some reason, QEMU doesn't provide a PS/2 mouse by default
lost schooner
that was the case
I was using the wrong bit from the config byte
anyway, now the issue is that the first port isn't responding correctly after a reset
lost schooner
that issue was because some of my defines for controller commands were wrong. one of which being WRITE_PORT_2
Going back to real hardware testing, the controller inits successfully now.
I get a ubsan type mismatch in printf whilst printing info about the first port
Thats fixed now
Now I am dealing with it freezing whilst doing something with the first initramfs file
lost schooner
Its always fun when you think things are working, but really they aren't and you just get lucky a few times in a row
I had some random freezes in the ps2 controller init, but they seem to happen only sometimes
Now, back to initramfs issues, it seems like the freezes occurs at some point in TempFileSystem::CreateFile
Not sure if it is in that function or something it calls, I just know that is roughly where the issue is
lost schooner
Wow. Its already been 3 days since a last gave an update. I haven't actually done anything in that time though
I've been thinking about taking all the VFS related stuff away from the kernel, and stress testing it in userspace with a bunch different sanitisers that GCC provides.
That might help track down the issue
lost schooner
I decided I might just try make a full userland port of my kernel
so far, it isn't proving to be that difficult
lost schooner
Most of the port isn't that bad. The annoying stuff is making correct calls to mmap/mprotect/munmap without having access to the linux headers.
dusky orbit
thats pretty cool, looking forward to seeing how this works out
I want to do something similar with my kernel at some point - have a version that runs in userspace for easy testing. My plan was to implement it as another 'arch'.
hardy lake
I decided I might just try make a full userland port of my kernel
tbh that sounds fun
I might make give my kernel an "x86_64-linux" arch
and then just port everything
(good thing I made OBOS portable)
lost schooner
well I got it to link
in a way
i still need to make the start function
which I've been avoiding
I just need to load the initramfs into memory, so it isn't that bad
to keep things simple, I might just use mmap to map the file
lost schooner
i ended up just creating a region with mmap, then reading the file to it, then close the file
I tried running it for the first time, and it seg faulted very early
lost schooner
well, the first major issue was that I didn't understand how linux program startup works
that is fixed now
now for some reason it wants to use some FS base thing (Thread local storage?), but I'm keeping thing freestanding, so that isn't set-up
maybe there is some flag to disable it
I know that it can be disabled at GCC configure time, but not at compile time
GCC has generated a bunch of mov rbx, QWORD PTR fs:0x28 instructions, but FS is 0
lost schooner
Oh ok
lost schooner
disable the stack protector
its off, but still 10 instances of that line of assembly
actually, there are 123 reads from QWORD PTR fs:0x28
I've turned the stack protector and ubsan off
I guess an option is to not be freestanding
I didn't really want to do that, but it seems like the only option
full needle
did you recompile everything after changing cflags
lost schooner
Yes. cmake is nice for making sure that happens
lost schooner
ok, I've decided that I will not stop being freestanding
its just going to be a pain to make things work
there must some documentation somewhere about what is in the struct which fs holds a pointer to
I tried giving GCC -mno-fsgsbase, which it seems to have ignored
I finally got it to not use fs/gs
I needed to give GCC a bunch of different flags
I had to do this:
-fstack-protector
-mno-fsgsbase
-mno-tls-direct-seg-refs
-mstack-protector-guard=global
now it doesn't seg fault immediately
it doesn't seg fault at all
mmap for the initramfs fails though
with error EINVAL
turns out the arguments are out of order
full needle
I had to do this:
```
-fstack-protector
-mno-fsgsbase
-mno-tls-direct-seg-refs
-...
you're turning the stack protector on?
lost schooner
you're turning the stack protector on?
yes, I got it working
just needed that last flag
full needle
ah fair
lost schooner
I did a bunch of refactoring to do all the linux syscalls from real assembly instead of using inline assembly, because it just wasn't going to work well
now it seg faults in kmalloc init
which is a bit further, but still not ideal
it works
its very fast
everything I have set-up so far works
time to actually have some fun with this
time to build it in normal mode to see if that still works
there is a bunch of extra warnings
and it deadlocks
something to do with my VMM
which doesn't make sense why it would suddenly break
it is disabled in the userspace port
its just really poor timing
I guess ive got something to debug
wait
how is this even possible
3 thread trying to acquire a lock, when the 4th thread hasn't even started yet
ahh
I commented out the global constructor stuff instead of wrapping it in an ifdef
it works
a bit unstable, but works
and very slow
lost schooner
in usermode, I've disabled a bunch of stuff to try keep it simple to start off with. The scheduler is fully disabled, the PMM is disabled, the VMM is replaced with a wrapper around mmap/mprotect/munmap syscalls. Its fully nongraphic, so kernel stdout/stderr redirects to real stdout, kernel stddebug redirects to real stderr. All drivers, and most arch specific stuff is disabled. The kernel heap is still functional, so is the VFS, symbol table loading (which doesn't actually get used at the moment), and the initramfs.
I still need to link kernel stdin to real stdin
which will be annoying to do properly
It might be possible to get a version of the scheduler working, but it will need to be really stripped down as everything runs in ring 3
if I actually get the scheduler working, it would just be fancy function calls
dusky orbit
ah yeah, that is lightweight, but its something
still cool
could you not implement stdin/stdout as a driver for a serial device?
or some analogue in your kernel
lost schooner
could you not implement stdin/stdout as a driver for a serial device?
maybe...
I might have to try making a real serial driver first so I can understand how it works
dusky orbit
its pretty straight-foward: a way to read data, write data and maybe setup an interrupt for when data is available to read.
lost schooner
I was just looking at the osdev wiki page, and it really does seem simple
lost schooner
maybe...
I've been thinking about this ^^^ some more, and I think I have worked out how I want to do it.
currently, everything is a massive mess (even before all this userland stuff)
direct calls to a VGA device from the file descriptor
direct calls to arch-specific debug (port 0xE9) writing functions
reads/writes to stdin/stdout go to the TTY which the file descriptor controls, but the seeking, goes directly to the backing VGA device
which is really bad
I was thinking on making it so TTYs have a backend, which is the only part that interacts with the raw underlying device
that way I can have a framebuffer backend for a graphic TTY
a serial backend for a serial TTY
or even a backend for port 0xE9 debug printing
I guess that means TTYs can have multiple backends for handling different stream types, such as stdin/stdout, stderr, and stddebug