this will be passed through a couple functions
#OBOS (not vibecoded)
flint idol
oh, and if a fault happened to do 2+ things for some reason
like it did a CoW thing
but also mapped part of a file
then the highest fault type will be recorded
typedef enum {
// A soft fault is when a swap in could use the dirty or standby lists to do a swap in, the part of the page cache at the file offset
// of the address was already filled in, or the fault was a CoW fault.
SOFT_FAULT,
// A hard fault is when a swap in needed to read from the swap_dev, or a part of the page cache needed to be read from disk
// to satisfy the fault.
HARD_FAULT,
// The page fault was caused by an access violation.
ACCESS_FAULT,
} fault_type;```luckily
most of my faults are soft faults
so they wouldn't take too much time
[ LOG ] Currently at 24372 KiB of committed memory (4452 KiB pageable), 8920 KiB paged out, and 19920 KiB non-paged, and 43380 KiB uncommitted. Page faulted 1926 times (147 hard, 1779 soft).
flint idol
#1061407633745125397 message
marker
so I might make a i386 breadboard "computer"
that runs obos
If I end up doing VGA, I'm using the SCN2674
Physical memory map:
00000000-1400000 20M: Usable, RAM
1400000-FFFF0000 4075.9M: Reserved, Unused, IO is dropped
FFFF0000-FFFFFFFF 64K: Reserved, ROM
Ports:
0x20-0x22: Intel 8259 PIC "Master"
0x40-0x44: Intel 8253/8254 "PIT" Ports
0xA0-0xA2: Intel 8259 PIC "Slave"
0x3F8-0x400: 16550 UART "COM1"
All IO on other ports is dropped
lgtm
lean glen
bro is copying my idea 😭
lean glen
btw you'll probs have to design a breakout board
flint idol
magic
as I said I might or might not do this
I saw on some reddit post that the i386ex might be better suited for breadboards
lean glen
I saw on some reddit post that the i386ex might be better suited for breadboards
that's because it can run at slower speeds
flint idol
and theorerical
lean glen
but afaict the 486 and 386 can run at slower speeds
atleast according to the guy making the series
flint idol
and hopefully
I blow nothing up
or give up on the project halfway though
or after I bought everything
lean glen
idk if there's any cpu with a mmu that's DIP
sorry, paging*
mmu there's the 286
nvm even the 286 isn't DIP
I guess you could do the m68k, you already ported your OS to it
though you won't have paging
yeah the only way to have a relatively modern CPU is with a breakout board
flint idol
I mean
weary hound
though you won't have paging
don't some 68k chips have an mmu built in?
flint idol
man ydo
weary hound
and exist in dip form
flint idol
but I assume they're not DIP
lean glen
don't some 68k chips have an mmu built in?
yeah but they arent DIP
I haven't found any
unless I'm stupid
flint idol
I guess you could do the m68k, you already ported your OS to it
I have my kernel ported to the m68040
weary hound
pretty sure you can also get an external mmu in a dip package?
flint idol
the mmu is different for each m68* chip
flint idol
pretty sure you can also get an external mmu in a dip package?
I think you can for older m68* chips
@devout niche knows more about this than me
lean glen
mhm yeah maybe 68020
could work with an external mmu
can you even find that in DIP tho
nah, but the 68010 could work
lean glen
with a MC68451
lean glen
both are
devout niche
the 68010 is a reasonable target
lean glen
68451 was meant to be used with the 010
flint idol
I wonder if I could use the same x86 PC hardware for this
devout niche
with the 68010 you would have to have a lite kernel as there's 16 mib of address space only
lean glen
"Some vendors used their own MMU designs, such as Sun Microsystems in their Sun-2 workstation and Convergent Technologies in the AT&T UNIX PC/3B1."
flint idol
with the 68010 you would have to have a lite kernel as there's 16 mib of address...
my kernel is pretty small in terms of binary size
and I can probably get rid of some components
lean glen
you could always just do the 386 thing
but it's a bit trickier since you need a breakout board
flint idol
my kernel after MMU init on x86-64 takes a total of ~6 mib
if I could agressively optimize for size
it could take less
(once I fix UB bugs ofc)
on the m68040 port it uses ~8M after vmm init
flint idol
my kernel after MMU init on x86-64 takes a total of ~6 mib
*VMM init
lean glen
I'd personally do 386/486
to get that 90s workstation thing
- designing a breakout board sounds fun
https://github.com/FoxCutter/486Dev/tree/main/486 Breakout the guy who did the series open sourced his
flint idol
found an i386 datasheet
flint idol
```
Physical memory map:
00000000-1400000 20M: Usable, RAM
1400000-FFFF0000 4075...
this might need to be changed based of the i386 pinout
it has 24 address lines
waiiiiit
if there's only D0-D15
how would 32-bit IO be done
I'll look at the other pins to get a hint
weary hound
how would 32-bit IO be done
two 16-bit accesses?
flint idol
I guess
I assume the top 8 bits of physical addresses are dropped
I think dma is done by first setting HOLD to high on the CPU then doing my memory access and setting HOLD to low after
this is all speculation
I'll read the manual
lean glen
you might want to build a simpler system first tho
flint idol
yeah maybe I'll start learning with a 6502
weary hound
you signal HOLD, wait for HLDA, do io, release HOLD
flint idol
and that
it'd be funny if I make SMP i386
but I won't
because I like my sanity
but I might write some acpi stuff if I have enough ROM
flint idol
yeah but it'd be cool getting uacpi running on this
lean glen
I think you're underestimating how complex ACPI is
you have to build all the tables yourself
flint idol
why build them when I can hardcode them
as you said, I already know my hardware
because ASL/AML is schizophrenic, I won't do that
actaulyl then what's the point
nvm
flint idol
ok so I have figured out IRQs probably
assert INTR
while setting D0-D7 to the interrupt vector
then after 2 bus cycles, the processor does its shit
When asserted, this input indicates a request for interrupt service, which can be masked by the Intel386
SX CPU Flag Register IF bit. When the Intel386 SX
Microprocessor responds to the INTR input, it performs two interrupt acknowledge bus cycles and, at
the end of the second, latches an 8-bit interrupt vector on D7–D0 to identify the source of the interrupt.
I assume if it's asserted, but eflags.if is zero, the processor queues the IRQ?
lean glen
another fun project could be fpga
flint idol
too expensive
lean glen
probs gonna be cheaper than a whole ass breadboard computer lol
flint idol
I mean...
the ICs used won't be too expensive at all
because of how old they are
or at least that's what I hope
the most expensive thing would be all the breadboards tbh
first fgpa I found is 150 cad
weary hound
too expensive
not that expensive if you get a cheap one :^)
flint idol
but where's the fun in getting a fgpa when I'll have to learn how it works
WAIT
implement the breadboard computer on an FGPA 
Physical memory map:
00000000-00C00000 12M: Usable, RAM
00C00000-00ff0000 3.9M: Reserved, Dropped
00ff0000-01000000 64K: Reserved, BIOS ROM```
hopefully this phys. memory map will workas long as it fits into 24 bits
Note the I/O addresses used by the
automatic I/O cycles for coprocessor communication are 8000F8H to 8000FFH,
if I want a coprocessor, I think I might need to work around that
maybe the m68010 would be simpler...
weary hound
i mean this seems simple enough?
weary hound
port i/o enable = port i/o cycle && !A₂₃
coproc enable = port i/o cycle && A₂₃
flint idol
I don't wanna do that
so yeah
I found an i386DX for 8 cad
on ebay
and an i386SX for the same price
16$ with shipping
and a coprocessor for 13.50$
I found a pit for 21.50$
for the pic, I'll I'm finding is intel xeon chips
:/
ok I found a PIC
for 12$
I found the 16550 UART
and those are the major parts
I hope
I'll use digikey to find SRAM and ROM
I can't find 12M SRAM
but I can find 16M
and 8M
so depending on the price
I will choose one
flint idol
whoopsie!
oberrow@Acer-AIO:~/Code/obos$ git diff --shortstat
48 files changed, 1548 insertions(+), 1875 deletions(-)```lmao
that was committed in the "I don't even know at this point" commit
flint idol
project ain't dead, just wanna work on the i386 breadboard thing instead
I'll work on both, trust
vale nymph
Obos is dead
flint idol
I'll work on obos rn (schematic drawing is getting boring)
after I get into userspace, I might work on an i386 port (for this purpose)
since I use the standard ports for the PIC/PIT/UART for my breadboard computer, I'll be fine on that regard
it's just the memory map that's a problem
*phyiscal memory map
I could just make my bios an ultraprotocol impl.
flint idol
anyway
I need to implement CoW
but this time no procrastination through discord
or youtube
or minecraft\
flint idol
if (rng->cow && (ec & PF_EC_RW))
{
// Mooooooooo.
fault_type curr_type = SOFT_FAULT;
if (curr_type > type && handled)
type = curr_type;
}```moooo
@crude parrot since you were interested in obos dev, here is my progress report thread
anyway, I will now work on symmetric CoW
then asymmetric cow
should the latter be used for private file mappings?
wait lemme re-read fadanoid's explanation
ignore that
so if I understood correctly
when you map a page as asym. cow, on read-fault you map it as shared, and on write-fault, you make yourself a new private page and copy the data
when you map a page as sym. cow, you map it as present, and it shares a page with the other cow page, until you write-fault on it, then if there are no more copiers, you steal the page, otherwise you make a new one and copy into it
flint idol
I hate page ranges
if I split a page range
what happens to the cow pages
actually
nvm that
what happens to the ref counts
and the referencers of the pages
what if I just don't split page ranges
and instead have a list of modified parts of the range
like say you mprot part of a region
but not all of it
then instead of splitting the region
it appends something to the region telling it that the prot at offset n was changed to
or even better
nvm
yeah so I could do that
it will also try its best to defragment the list
it also saves memory as a bonus
so instead of having three regions when you split a region in the middle
you have one, but the middle of the region has a node in the list saying that it was modified this and that way
struct modified_region_node
{
struct modified_region_node *next, *prev;
uintptr_t offset;
size_t size;
enum {
PROT_CHANGE,
UNMAPPED,
} type;
union {
page_protection new_prot;
bool is_unmapped;
} un;
};```then a page_region has a list of these
and when something goes to remap the page (e.g., a swap in operation), to get the protection, it will search this list, and if it finds that page, it uses node.un.prot, otherwise it uses reg.prot
but before I do all that
flint idol
and now I'm adding a struct page (but for physical pages)
which will delay the commit by a day or two
or three
or four
who knows 
vale nymph
wait I thought you were already doing that
how r you handling stuff like dirty or standby lists
flint idol
Yes.
flint idol
wait I thought you were already doing that
haven't had it in any of my kernels
vale nymph
damn
flint idol
typedef struct idle_page_node
{
uintptr_t virt;
uintptr_t phys;
page_protection* prot;
struct idle_page_node *next, *prev;
} idle_page_node;```that's for dirty/standby lists
vale nymph
what if two different processes have written to the file in memory
does it go there two times how does that work
struct page per virtual page is defo an interesting choice
flint idol
what if two different processes have written to the file in memory
yes I thought of that
that's a bug
which I was going to fix
thick jolt
hi
flint idol
so uhh
time to make a struct page
typedef struct page
{
uintptr_t phys;
size_t refcount;
enum {
PHYS_PAGE_STANDBY,
PHYS_PAGE_DIRTY,
PHYS_PAGE_USED,
} status;
bool huge;
} page;```I think that's all I neeD?
*need
then it's allocated using the vmm allocator
and stored in an rb-tree
because we love our rb-trees don't we
as soon as its refcount goes to zero, the underlying physical page gets freed
flint idol
I probably (hopefully) don't need to change much
I need to change:
- File mapping
- Swap In/Out
- Standby/Dirty
- CoW (not implemented, so that's fine)
I'll start with swap
so I need to slightly change the swap interface
to take a physical page
as the key
or really
a struct page
which then uses the phys member as the key
typedef struct swap_device
{
// pg->phys should be the key
obos_status(* swap_resv)(struct swap_device* dev, page* pg);
obos_status(* swap_free)(struct swap_device* dev, page* pg);
obos_status(*swap_write)(struct swap_device* dev, page* pg);
obos_status(* swap_read)(struct swap_device* dev, page* pg);
obos_status(*deinit_dev)(struct swap_device* dev);
void* metadata;
} swap_dev;```this is the new swap interface
lol
gotta keep the streak
flint idol
ah shit
flint idol
it's clangd not liking gcc attributes and cmd line parameters
flint idol
ah shit
how do I lookup the phys page tree for a virtual address on swap in, when I don't have the physical address
because the page is paged out
flint idol
because the page is paged out
which means it's unmapped
I could just keep the phys address of a pte when it's unmapped
and fill that in on query
but that has a flaw
if the pml2+ is empty, the kernel frees them
so I can't do that
I could have some struct
that has a page* and a list of virtual pages related to it
that's used for standby/dirty lists
lean glen
how do I lookup the phys page tree for a virtual address on swap in, when I don'...
Can't you get the paddr from the pfn
flint idol
don't have that
lean glen
Bruh
flint idol
another day, another time wayland caused chrome input to stop working
ok I'm going to be honest, I have no idea what to do here
flint idol
that has a page\* and a list of virtual pages related to it
this is bad
because it's slow
if there are many pages CoWing a physical page
wait
nvm
flint idol
if the pml2+ is empty, the kernel frees them
I'll just add a flag to tell the arch to not free any PTEs
and to only clear present
flint idol
ok so
I have a problem (that I solved, just documenting it)
if I swap in a page that's on standby/dirty lists
then we're good
otherwise it needs to ask the swap file what that page consists of
but the swap file interface thingy
takes in a physical address as a key
but that's a problem because the page would've been reused since it's not dirty or standby
so my solution
typedef struct swap_device
{
// *id needs to be aligned to OBOS_PAGE_SIZE if !huge_page, otherwise it needs to be aligned to OBOS_HUGE_PAGE_SIZE
obos_status(* swap_resv)(struct swap_device* dev, uintptr_t* id, uintptr_t phys, bool huge_page, size_t refcount);
obos_status(* swap_free)(struct swap_device* dev, uintptr_t* id);
obos_status(*swap_write)(struct swap_device* dev, uintptr_t* id, page* pg);
obos_status(* swap_read)(struct swap_device* dev, uintptr_t* id, page* pg);
obos_status(*deinit_dev)(struct swap_device* dev);
void* metadata;
} swap_dev;```is to change the swap interface thing to that (as opposed to everything taking a struct page* pg)
then to store a list of virtual pages in dirty list nodes
and whenever the dirty page writer gets to that node
it sets the physical address of the virtual pages to id
flint idol
and whenever the dirty page writer gets to that node
actually
it will set a field in struct page to the id
then when a standby page is taken
flint idol
it sets the physical address of the virtual pages to `id`
it does this
it will also set one of the kernel-reserved bits of the pte to true to indicate that the physical address is actually a swap id
and that the kernel needs to read from swap
flint idol
```c
typedef struct swap_device
{
// *id needs to be aligned to OBOS_PAGE_SI...
refcount will be set to the amount of virtual pages that will be using the swap space
and on swap_free, it will only actually free the swap space when !(--refcount)
limber wave
Obos is dead
that's gooooood. Now oberrow can start a new rewrite! (Dark Souls level pain)
limber wave
you know those people who beat every Souls game no damage one after another? You are trying to do that with OSDev or?
flint idol
I don't know em
limber wave
DS frustration >>>>> oberrow curse
flint idol
bruh I've wasted so much time
today
I got a total of 10 loc for obos
done
and a broken debian install
well it's not that broken
boot is just a bit scuffed
limber wave
That's my daily reminder that Linux sometimes sucks
flint idol
the problem is me
I misconfigured some firmware setting
to be IDE
to make my drives IDE
and when I change it back to ahci linux gives up on root mounting
empty kernel
and a broken debian install
what did you do to it this time
flint idol
what did you do to it this time
/var partition
decided to delete it
and move it to the root partitin
*partition
without saving the data on it
empty kernel
why not just have only a root partition and an efi partition to start with
no fancy var or home partitions
flint idol
idk why I didn't do that
I do home partition because I sometimes fuck up the partition table
and if I did dd on sda by accident, I would at least be able to recover my /home partition
since it's well into the disk
empty kernel
I do home partition because I sometimes fuck up the partition table
that makes sense
flint idol
it's 35G into the disk
so if I dd any iso by accident I can use my fancy program that looks for the ext4 signature
empty kernel
how many partitions do you have?
flint idol
and then it'll be able to recover most if not all data on /home
flint idol
how many partitions do you have?
6
one is a ms reserved
one is the windows C: drive
a swap partition
a root partition
and the efi partition
and the home partition ofc
empty kernel
wait, I have 6. Mine are different though. EFI, MS reserved, win C:, win recovery, arch root, arch swap
flint idol
if you install windows after linux you'd end up somewhat with what I have
flint idol
hm?
empty kernel
I used diskpart to create the partitions, then used dism to apply the image, followed by bcdboot for the boot files
flint idol
ok
empty kernel
oh and some magic to get the recovery partition to work
flint idol
```c
typedef struct swap_device
{
// *id needs to be aligned to OBOS_PAGE_SI...
idk why I added the phys parameter to swap_resv
unneeded, and I have no idea what to do with it
flint idol
and that the kernel needs to read from swap
I just realized there is a problem with this
if there is only one sharer to a physical page and it gets swapped out, all is good
but if there are multiple sharers and the page gets plucked from standby
when the kernel goes to swap in the pages, each get their own copy
as they'll start modifying a different physical address each time
I can't be putting too much stuff onto the swap device, as that can get slow FAST
specifically on disk where it needs to read each time
maybe if it's in-ram swap it's fine
flint idol
as they'll start modifying a different physical address each time
I guess
ugh
I could keep a hashmap or smth that keeps track of what swap allocation IDs belong to what virtual pages
but that takes memory
but that's all I could think of rn
flint idol
I could keep a hashmap or smth that keeps track of what swap allocation IDs belo...
basically it'd be like:
struct thingy
{
page* pg; // if nullptr, it must be read, otherwise it can be stolen.
size_t refs; // when !(--refs), free id and this structure, but do not free `pg`
uintptr_t id; // key
};```I think I will be using a linked list to store these
since I don't think I'll be hard page faulting often enough to need really fast lookup on it
flint idol
so
theoretically
it should kinda boot
it gets to pnp
then crashes
and it boots
after a quick bug fix
well it boots
but not correctly
and also reports zero page faults
because it doesn't page fault...
weird
something tells me my unmap function isn't working
found it
now ti hangs
because of a deadlock
something forgot to release a lock
nope
just recursive page faults
not recursive
but instead
incorrectly handled I guess you could say
ok
now I use after free in uacpi
in phys_page_cmp
ez fix
now it boots
[ LOG ] Currently at 24560 KiB of committed memory (4452 KiB pageable), 9384 KiB paged out, and 20108 KiB non-paged, and 18014398509459052 KiB uncommitted. Page faulted 2044 times (148 hard, 1896 soft).```not that much uncommitted memory tbh
[ LOG ] Currently at 24432 KiB of committed memory (4452 KiB pageable), 9384 KiB paged out, and 19980 KiB non-paged, and 184 KiB uncommitted. Page faulted 2044 times (148 hard, 1896 soft).```fixed that
so I took the test driver that had the fireworks test
modified it a bit to call the test
and the fireworks part is barely happening
sadly
and after a bit
you use after free
uhhh
just a bit of memory corruption here and there
because obos wouldn't be obos without that, amiright
core event set
signal waiting threads
thread list remove
unaligned access
damn it
that's the node being passed
something tells me, it ain't legit
it took a memset to fix it
these look a lot smoother now idk why
maybe the vmm was a big bottleneck before
flint idol
you use after free
I think I have the bug fix to this
well it doesn't use after free anymore
instead this:
Assertion failed in function Mm_SwapOut. File: /home/oberrow/Code/obos/src/oboskrnl/mm/swap.c, line 50. page->range->pageable
oh nvm it is use after free
but it looks kinda funny
flint idol
I fixed the use after free
and will now implement CoW
I feel like I should look into using a code formatter for obos
flint idol
when you map a page as sym. cow, you map it as present, and it shares a page wit...
marker
flint idol
so private file mapping nearly works
vale nymph
r you mapping directly from the page cache
flint idol
yes
vale nymph
epic
flint idol
oh nvm private file mapping does work
there are locking shenanigans
oh nvm it's not locking stuff
ok NOW it's locking shenanigans
or maybe not
why is a non-pageable page in the ws
vale nymph
Oberrow curse
flint idol
I have:
- use after frees
- non-pageable pages in the ws
- memory corruption (seemingly in a thread's context info)
atm
I think I know why
there is a non-pageable page in the ws
yup that was it
now instead of that
I get a pf in its place
but that was expected
I feel like it's a bug with the ahci driver
or rather, a regression
flint idol
bro wth is this register dump
check_exception old: 0xffffffff new 0xd
106561: v=0d e=0000 i=0 cpl=0 IP=0008:ffffffff8002da06 pc=ffffffff8002da06 SP=0010:ffffff00004bcc58 env->regs[R_EAX]=2020202020202030
RAX=2020202020202030 RBX=2020202020202020 RCX=0000000000000000 RDX=2020202020202020
RSI=0000000000000010 RDI=00000000000001c0 RBP=ffffff00004bcd18 RSP=ffffff00004bcc58
R8 =0000000000000000 R9 =0000000000010000 R10=0000000042680000 R11=0000000000000000
R12=ffffffffffffeed8 R13=0000000000000000 R14=0000000000000000 R15=0000000000000000
RIP=ffffffff8002da06 RFL=00200207 [-----PC] CPL=0 II=0 A20=1 SMM=0 HLT=0```bro what is 0x2020202020202030
this looks like a buncha spaces then another character in ascii
or rather the other way around if this value was loaded from ram
flint idol
note to self: update packages when you see this
(that has nothing to do with obos, I was just reminding myself tomorrow when I wake up and look in here)
real pecan
Should I delete his reminder
empty kernel
It would be funny
hazy socket
when did you start this os?
flint idol
Should I delete his reminder
I remembered to do so myself
flint idol
when did you start this os?
May 2023
Although this rewrite started in may of this year
hazy socket
Wow
flint idol
bro wth is this register dump
only solution is to rebuild and hope for the best
didn't work
but for some weird reason
this doesn't happen when obos.iso is a cdrom
flint idol
this doesn't happen when obos.iso is a cdrom
I feel like it's the ahci driver once again
when I stop the firework test
and make obos.iso be a drive
it crashes
when probing sda (obos.iso) for partitions
FUCK AHCI
thank you for coming to my ted talk
ahci_start_dma ahci(0x55758d881870)[0]: start dma
ahci_populate_sglist ahci(0x55758d881870)[0]
ahci_dma_prepare_buf ahci(0x55758d881870)[0]: prepare buf limit=4096 prepared=4096
ide_dma_cb IDEState 0x55758d883090; sector_num=1 n=8 cmd=DMA READ
ahci_cmd_done ahci(0x55758d881870)[0]: cmd done
ahci_trigger_irq ahci(0x55758d881870)[0]: trigger irq +DHRS (0x00000001); irqstat: 0x00000000 --> 0x00000001; effective: 0x00000001
ahci_check_irq ahci(0x55758d881870): check irq 0x00000000 --> 0x00000001
ahci_irq_raise ahci(0x55758d881870): raise irq```ahci_port_write ahci(0x563453427960)[0]: port write [reg:PxCI] @ 0x38: 0x00000001
ide_bus_exec_cmd IDE exec cmd: bus 0x5634534290f8; state 0x563453429180; cmd 0x25
ahci_start_dma ahci(0x563453427960)[0]: start dma
ahci_populate_sglist ahci(0x563453427960)[0]
ahci_dma_prepare_buf ahci(0x563453427960)[0]: prepare buf limit=31744 prepared=28672
ahci_cmd_done ahci(0x563453427960)[0]: cmd done
ahci_trigger_irq ahci(0x563453427960)[0]: trigger irq +DHRS (0x00000001); irqstat: 0x00000000 --> 0x00000001; effective: 0x00000001
ahci_check_irq ahci(0x563453427960): check irq 0x00000000 --> 0x00000001
ahci_irq_raise ahci(0x563453427960): raise irq```not this again
inland radish
```
ahci_port_write ahci(0x563453427960)[0]: port write [reg:PxCI] @ 0x38: 0x000...
is this the first mass storage driver for obos?
flint idol
since the third kernel yes
the third kernel had an ahci driver, but that one was pretty shit
flint idol
```
ahci_port_write ahci(0x563453427960)[0]: port write [reg:PxCI] @ 0x38: 0x000...
limit is the bytes needed, and prepared is the bytes in the PRDT regions
I'm missing 3072 bytes
vale nymph
why not nvme, its simpler than ahci
flint idol
two reasons
- I already understood ahci
- I have no NVME test subjects
flint idol
I have a bug with FAT12 in slowfat
an unaligned access
fixed
but now there's a bug
in FAT12
where the filenames don't exist
the directory tree is intact, but nothing has a filename
fixed
if (lfn_entry_count)```
Was used instead of
```c
if (*lfn_entry_count)
and lfn_entry_count is a pointer
lean glen
I also have no machines with working nvme
a ramfs is better for testing anyway
but on VMs disk is much faster
haughty abyss
ramfs?
vale nymph
but on VMs disk is much faster
yeah it sucked having to wait for it all to load
implementing proper file caching and a disk driver was much better
flint idol
obos has no tmpfs, only initrd (where every file is RO), and a FAT driver
vale nymph
thats only useful for testing a vfs though
tmpfs has some more uses
like /dev/shm on linux is a tmpfs
lean glen
thats only useful for testing a vfs though
Nah on real hardware for example
haughty abyss
tmpfs has some more uses
well yeah true
lean glen
Where you can't install to disk
vale nymph
usb mass storage
haughty abyss
just do ramdisk.qcow2.lzw :^)
lean glen
just do ramdisk.qcow2.lzw :^)
Yeah lz4 compression is probs the best tradeoff for the ramdisk
lean glen
We came to that conclusion when testing my tarmac thing
flint idol
amazing, obos doesn't crash with optimizations on
the kernel is only 459K
and that's with uacpi compiled into it
and it runs many faster
the best part is
the vmm only PFs 838 times
vale nymph
astral is 5 mb cuz I compile with debug and dont strip symbols
flint idol
the thing is being funny
lean glen
astral is 5 mb cuz I compile with debug and dont strip symbols
I have no clue why mine is so big then
flint idol
where did I add ubsan.c and kasan.c to my kernel sources
flint idol
bruh why does some random event object always have 0x1 as it's head of waiting objects
causing a pf whenever that object is signalled
but only when there is an ahci drive
fun fact: the last address in the kernel binary belongs to uacpi_unwrap_internal_reference
flint idol
fun fact: the last address in the kernel binary belongs to `uacpi_unwrap_interna...
*in the .text section
I will now implement asym CoW
and hopefully that's all I needed for the vmm rewrite?
inland radish
two reasons
- I already understood ahci
- I have no NVME test subjects
i only had my old PC and my laptop
as real hardware test subjects
flint idol
when you map a page as asym. cow, on read-fault you map it as shared, and on wri...
@devout niche did I understand asymmetric CoW properly?
I'm gonna port my fireworks test to linux
devout niche
<@391791686881247234> did I understand asymmetric CoW properly?
Yes at quick read that sounds fine
flint idol
hmm ok ty
flint idol
I'm gonna port my fireworks test to linux
it'd be fun if I could use the alpha component for this
https://www.kernel.org/doc/Documentation/fb/api.txt
this barely makes sense to me
but I think I need FB_VISUAL_TRUECOLOR
nvm I just needa rtfm better
inland radish
what if I just don't split page ranges <:galaxybrain:701707619651420251>
i would split
flint idol
I do
inland radish
Yes at quick read that sounds fine
there was some kind of problem with symmetric cow wasnt there
flint idol
I proposed that outta confusion
inland radish
when the cow page is paged out or something
flint idol
are you talking about my kernel or in general?
inland radish
i'm talking about in general
flint idol
nvm I just needa rtfm better
So it works, but it looks funny
I was using usleep
For milliseconds
But it seems like my random numbers aren't very random, or I'm doing something wrong
flint idol
well I tried fixing that and now it stopped working
here's the source
@lean glen I did port my fireworks test to linux
and it looks pretty good
lean glen
how do you test it
flint idol
just idk how good my /dev/fb0 handling is
lean glen
vm?
flint idol
as in, I ported it to userspace
lean glen
also you couldnt put it in a single file 😭
lean glen
omfg you're using inline asm
lean glen
you do know rand() exists?
flint idol
yes
lean glen
as in, I ported it to userspace
yeah but how did you test fbdev
did you test it in your tty
flint idol
yes
lean glen
cool
flint idol
you do know rand() exists?
iirc I didn't like the results with rand()
flint idol
oh I thought you meant using rand as the rng
flint idol
@lean glen I pushed the code for the linux fireworks to github if you care
makefile was rushed, don't judge
flint idol
bruh my vscode broke
it's basically frozen loading a cmake preset
it took like 20 seconds
or even a minute
I should have asymmetric CoW implemented now
I'll be testing it by having a zero page that's handed out for anon mappings
might be interesting to see if I can use this for kasan
to catch uninited memory use
vale nymph
obos curse
flint idol
ok thank god, I fixed it
I was accidentally returning from the asym CoW handler before remapping the page to the new physical address
flint idol
obos fireworks test now deadlocks after ~1 minute of running
vale nymph
when will obos turn my computer into a firework show tho
flint idol
when will obos turn my computer into a firework show tho
coming soon™️ to a github near you
bruh
the thread that has the context lock is simply in yield
but has no back trace
oh it's the current thread
so it's a recursive lock of a context lock within the pf handler
it was a bug with the vmm init
specifically in my round_up macro
wonder if I could make my fireworks test a screensaver
since I already have it ported to linux
vale nymph
lgtm
flint idol
so I have CoW
that's all I needed, right?
well it boots, so fuck it
I'm committing
Mm: Small changes
so when I left off, I was doing signals
which I finished
so it's TTY time
devout niche
https://github.com/OBOS-dev/obos/commit/027f2d4e7990a5bb4594f7857f4eeeb4af71c4ba...
remarkable work
flint idol
Ty
I still need to implement one last thing which is on disk swap
But since it was already a huge commit, I refrained
And I also want to make more of the kernel pageable
*kernel code
Like the VFS memory for vnodes and stuff is pageable, however the code is not
Also will try to abstract vnode reads
Because it takes quite a few lines to actually get the driver that a vnode uses
flint idol
No manual entry for tty in section 9
flint idol
surely if I just implement the functions needed for a tty, I'll be fine...
these
from man termios
and the ioctls
from man ioctl_tty.2
idek if I need the ioctls...
I feel like my tty implementation will be shite
my brain is rotting reading tty ioctls
vale nymph
maybe start with ttys demistified first
flint idol
but in accordance with the UNIX design philosophy, applications should be kept as simple as possible
Fuck you unix
torpid wigeon
🤓
flint idol
maybe start with ttys demistified first
what I got from that:
line discipline (idk what the line did)
pts
tty
poor future me, I'll be reading my summary of tty demystified and that's all I'll see
I think pts
is just input/output buffer
flint idol
uhhh I don't think my fireworks test is supposed to run better single core
I fixed the bug with a single core where it would deadlock locking the global lock in uacpi
it was unzeroed memory
it seems like there is a bug with unzeroed memory that causes pfs that shouldn't crash to crash
nah fuck ttys, I need to deshittify my scheduler
there are 1521 threads running
with <100 particles
flint idol
it seems like there is a bug with unzeroed memory that causes pfs that shouldn't...
it is in the allocator
obos challenge: don't have an allocator bug
challenge level: locked
flint idol
I am genuinely confused
HOW
the debugger hits barely any of my breakpoints in the pf handler
but the kernel served 2129 page faults?\
printf my beloved
after some time of the fireworks test running the page fault count becomes lower and lower until it stops page faulting]
which I guess is an indication of how good my page replacement algo is
flint idol
it seems like there is a bug with unzeroed memory that causes pfs that shouldn't...
the region is unmapped
and was freed through the vma's free function
flint idol
it seems to have be an allocator mismatch
i.e., I was allocating using the kernel allocator, but freeing using the non paged pool allocator
it indeed was that
I added a way for kasan to identify when you allocate something, but free it with a different allocator
basically how it works is it stores the allocator used in a region, and on free, it checks if region->owner != This
and if that's true, then it reports an asan violation
I wonder how long I can leave the fireworks test running
WTF IS THIS MEMORY LEAK
I swear it allocated like 4M in like 5 seconds
ok it's not leaking
just a really chunky stack
yeah it's ~8M worth of stack for a firework
each thread has a 0x10000 byte stack
at max it can be 12.5M for a firework
but there is a memory leak
somewhere
sample one: 36331520 bytes committed
sample two: 36474880 bytes committed
both samples were taken at an idle period
the leak could be in the:
- timer interface
- irq interface
- scheduler
- (unlikely) vmm
- fireworks test itself
the timer interface does not allocate memory
the irq interface sounds unlikely
there was one in the scheduler
if (!currentThread->references && currentThread->free)
currentThread->free(currentThread);```I forgot to decrement the reference count
sample one: 29376512
whoops
wtf
wtf
why is Arch_IRQHandlers corrupted?
specifically when I increment the refcount of a thread in the spinlock acquire function
so I could see what thread is forgetting to release the spinlock
but it only happens when I free the current thread in exit current thread
I finally got to deferring the freeing of scheduler structures to a DPC (maybe useless since exit current thread and DPCs run at the same irql)
holy memory leak
nvm
still big memory leak
flint idol
I need to fix memory leak
I will just search for each instance of ZeroAllocate
and find which instances aren't global
i.e., are a temporary kernel object
there is a big leak in Core_WaitOnObjects
but that isn't used anywhere as of now
what if it's just allocator fragmentation
vale nymph
Just run defrag ez
flint idol
omg how is the VMM leaking this much memory
there are huge blocks of memory allocated
that are never freed
the kernel allocator has a lot of allocations, but half of them were freed
note that all Mm_Allocator allocations are temporary, and only exist from the time the vma allocate function is called until the vma free function is called
flint idol
there are apparently no leaked page objects
zero page_node objects leaked
as for working_set_node
there are 2000 alive
idk how many are leaked
I've added checks for working set nodes (not entries)
and none are leaked per-se
they are kept alive by the working_set_entry that's allocated
I fixed most of the working_set_entry leaks
flint idol
I fixed most of the `working_set_entry` leaks
I think the rest are just working set nodes awaiting a free on the next sample interval
anyway there are mysterious objects allocated with the same size as a page range
but aren't a page range
flint idol
wtf is this shit
bro how tf is this freed and still in the rb-tree
this struct page*
I kid you not the only place those are freed are in their deref function
and before the free it removes the node from the tree
and in one other place
but that removes the node from the rb-tree
there was one other place that was not supposed to be freeing it
flint idol
I really need a blazing fast, memory safe language for obos
because quite clearly I don't know how to do memory safety
like my kernel is so cursed that kasan is scared of my kernel
Assertion failed in function Allocate. File: /home/oberrow/Code/obos/src/oboskrnl/allocators/basic_allocator.c, line 251. n != n->next```I have decided anyone saying that they don't need kasan
is lying
ok fuck this shit
will the obos curse get the curse get worse as halloween approaches
umbral iceBOT
flint idol
I have a use-after-free
caused by an allocator bug
obos classic
it only happens with kasan
obvious thing to do here is to drop kasan support
I most drop obos support for obos
since it crashes on x86-64 and m68k
flint idol
vale nymph
0 days since last obos memory bug
empty kernel
It's his curse
flint idol
guess the bug I'm debugging based on my status
I think I might implement something close enough to what I think a slab allocator is for the vmm allocator
(not vma, the vmm heap)
I'll have a max slab size of 256 bytes for the vmm allocator
#define SLAB_GET_NODE(reg, index) \
index < (reg)->nSlabs ? \
&((slab_node*)(((uintptr_t)((reg) + 1) + (reg)->slabSize - 1) & ((reg)->slabSize - 1)))[(index)] : \
nullptr```we just love the code readability for this macro
flint idol
I have discovered that I want a pushlock
to protect my linked lists
but the problem is
I don't have pushlocks
and I have no idea where to start
ty wikipedia
I'll be taking that
typedef struct pushlock
{
struct waitable_header hdr; // for writers
_Atomic(size_t) nReaders; // if > 0, wait on hdr, when it gets to zero, signal hdr.
thread* currWriter; // if == nullptr, no one is writing
} pushlock;```if I were to rewrite obos, I would actually use locks other than spinlocks
I implemented other sync primitives so late
that none were used
flint idol
omg linus torvalds contributed to obos
now time to revert the commit before he gets mad and DMCA take downs obos for my minor tomfoolery
limber wave
unkempt pawn
now time to revert the commit before he gets mad and DMCA take downs obos for my...
Linus® Torvalds™ is a registered trademark! How dare you!
flint idol
@frigid plover just come work on obos instead of making a community os 
flint idol
so after some bug fixes
the slab allocator nearly™️ works
Assertion failed in function Core_WaitOnObject. File: /home/oberrow/Code/obos/src/oboskrnl/locks/wait.c, line 28. Core_GetIrql() <= IRQL_DISPATCH```I fixed that
empty kernel
are you making a separate allocator type for the VMM heap?
flint idol
I'm writing a slab allocator for the vmm heap since it sounds more appropriate
empty kernel
it probably is
flint idol
since you won't be allocating objects of unknown size
max that can be allocated is probably ~512 bytes
idr
empty kernel
I didn't bother in my kernel since I didn't know what size objects would be needed
flint idol
memory corruption
if I were to rewrite obos I would make scheduler primitives as soon as I finish making the scheduler
I won't rewrite though, trust
this is how I feel rn
flint idol
Page fault at 0xffffffff80007d15 in kernel-mode while to read page at 0x000000003e17f060, which is unpresent. Error code: 0```bruh
in the kernel allocator something corrupts a region
thick jolt
flint idol
```
Page fault at 0xffffffff80007d15 in kernel-mode while to read page at 0x0000...
it has been 4 hours
plus yesterday's debugging
and I am still not one bit closer
to fixing this
I think I might git reset --hard HEAD and try this again
flint idol
commit spoofing lol
git config user.name "Linus Torvalds"
git config user.email "[email protected]"```dropping the slab allocator
flint idol
don't mind me
just pissed off
OBOS challenge: don't have a memory corruption bug
level: literally fucking impossible
fuck it
I'm rewriting the allocator
actually no
I'm rewriting the kernel /j
flint idol
OH AND OF COURSE IT FUCKING DEADLOCKS TOO
or is it just slow
it was a linked list being corrupted
maybe my kernel does need a redo.......
flint idol
I mean
empty kernel
Maybe try testing things without smp first, then once they work, try with smp
empty kernel
Without smp, there are less factors to deal with
flint idol
I got slightly distracted and can now time out drivers
that are hanging
idk if I should block the thread though...
real pecan
How do u know they're hanging
flint idol
well I busy loop as of now while waiting for drivers to finish their init
if (CoreS_GetTimerTick() >= file->id->main_thread_deadline)
{
if (!file->id->main_thread_timed_out)
{
OBOS_Warning(
"Waiting for driver %d (%s) timed out after %d MS (%dS)\n",
file->id->id, uacpi_strnlen(file->hdr->driverName, 64) ? file->hdr->driverName : "Unknown",
file->id->main_thread_timeout, file->id->main_thread_timeout/1000
);
CoreH_ThreadBlock(file->main, false);
}
file->id->main_thread_timed_out = true;
continue;
}```so I can do that
real pecan
Maybe make it drivers job to not hang
vale nymph
^^
flint idol
well I don't know what kinda mutex drivers hold
if it hangs inside of kernel code with a mutex held
real pecan
What if u abort it while it’s in the middle of e.g. Malloc code
This is incredibly unsafe
The thing u did
It has the same problem as signals in userspace
flint idol
if it hangs inside of kernel code with a mutex held
then that's kinda a kernel bug
I added an option to the driver header to disable timing out
real pecan
Drivers Malloc on init
vale nymph
what if it just takes a bit to get to where it needs but then malloc
say it was waiting for the device to respond or w/e
real pecan
what if it just takes a bit to get to where it needs but then malloc
Exactly
flint idol
maybe I can add some sorta sync primitive to the driver id
to tell the kernel to not time it out
because the driver know it's going to take a bit
real pecan
Why is this mechanism needed at all lol
real pecan
Make waits limited in time instead of relying on external management to kill you
flint idol
I guess that's a better idea
flint idol
funny thing
real pecan
From driver init
flint idol
my drivers don't have anyway to tell the kernel what's up with driver init
but anyway
real pecan
But why
Cant u make it return stuff from the entrypoint
Or like from the probe callback
flint idol
Or like from the probe callback
Probe callback is to probe an fs on a partition
flint idol
Cant u make it return stuff from the entrypoint
It starts a new thread
For each new driver
real pecan
It starts a new thread
Make a trampoline?
That calls the entrypoint and listens to the exit code
flint idol
meh
I mean I could
but like
how would I respond to the exit code
unload the driver?
flint idol
I guess I could do that...
I'll have a struct
struct driver_exit_code
{
obos_status status;
const char* context; // an additional message to print describing the failure. can be nullptr
bool fatal;
};```and have that returned from a driver
entry
real pecan
Struct name is weird but other than that fine ig
real pecan
Depends if u want to use this struct for more things
But something like driver_init_result or status
flint idol
that sounds better
typedef struct driver_init_status
{
obos_status status;
const char* context; // an additional message to print describing the failure. can be nullptr
bool fatal;
} driver_init_status;
void Drv_AbortDriverInit(driver_id* id, const driver_init_status* status);```Wait obos_status is a thing?
real pecan
I literally made it up there lol
flint idol
lol
real pecan
Guess I was right
flint idol
I was like "how does he know how I write my status codes 
"
real pecan
Lmao
flint idol
```c
typedef struct driver_init_status
{
obos_status status;
const char*...
I think I'll call the function Drv_ExitDriver
real pecan
That sounds like not something u call in case of an erorr
Also does your init thread never exit?
flint idol
well you can call it on success too
flint idol
Also does your init thread never exit?
kernel init thread or driver main threads?
real pecan
Driver
real pecan
So WTH do they do after init is done