NeurOS (Operating System Project)

[olive rivet]

This thread will track my progress towards developing an operating system themed around the Neuroverse. It's also a continuation of the discussion that was started in #1199825454027251784.

NeurOS is a project that I started on 1/1/2024 with the goal of learning about the intricacies of developing an operating system. It was also an opportunity to start learning about the Rust programming language. So far, I've been able to accomplish a lot in a short span of time. The operating system is now able to boot successfully on x86_64 machines that have BIOS or UEFI firmware. I've also been able to implement a few basic building blocks that will help shape the foundation of the operating system.

The video attached in this thread showcases the current state of the operating system. Some of the demonstrated features include font rendering, image rendering, logging, interrupt handling, and the linear framebuffer driver. These functionalities are currently provided by the kernel. They will eventually be provided by user space processes once I've implemented process scheduling, memory management, and inter-process communication (IPC).

The eventual goal is to provide a basic command-line interface (CLI) for interacting with the operating system. Down the road, this will lead into a graphical user interface (GUI) in the form of a desktop being created. I also hope to be able to port existing software to the system. At the very least, I aim to develop a POSIX compatibility layer that will allow me to port software from Unix-like systems (including DOOM).

Finally, if Vedal is interested, I'd be willing to collaborate to develop a mechanism that will allow Neuro-sama to interact with the operating system. I think that would be an amazing feat and would help to provide new content for the stream.

The source code for the project is located here: https://github.com/Theomund/NeurOS

Please let me know if you have any comments, questions, or suggestions.

[vapid lava]

Are you going t o make a Windows compatibility layer at some point

Also I think once you add a gui, if it has windows, they should match the ones in Neuro's starting soon background, like the store tiles in the Cookie clicker mod

[vapid lava]

Also how are you going to handle drivers? Are there going to just be built-in generic drivers or actual hardware-specific ones like most OSs

[vapid lava]

Preferably for drivers you'd have generic drivers for things that don't have device specific ones, and then device specific ones

[halcyon prism]

I understand correctly, are you trying to make an OS from scratch or a Linux or Unix add-on?

[vapid lava]

Based on what I've understood, (I usually understand this kind of stuff) this is an OS build from scratch

[halcyon prism]

It would be really hard, you know?

[vapid lava]

Some people like hard things

[halcyon prism]

I like your attitude)

[polar apex]

it does seem to be built entirely from scratch, or at least at a very low layer, given they're programming in Rust

[olive rivet]

I'm developing it from scratch.

[halcyon prism]

this is your first time doing this?

[polar apex]

is something like a driver compatibility layer a possibility, basically a driver for drivers? since in that case, one could just use drivers which already exist for a different OS

[vapid lava]

I guess that could work, since compatibility layers for programs from other OSs exist on existing OSs

[halcyon prism]

no, this is already an add-on over other systems or copying the architecture

if you don’t mind copying the architecture, then yes, but then it’s easier to make an add-on

[halcyon prism]

here I am talking about distribution

[olive rivet]

Device drivers are going to be daemon programs that run in user space. I think there'll be a mixture of both generic drivers (e.g. SATA, NVMe) and hardware-specific drivers. It's mostly going to depend on what I can test in a virtual machine and with the bare metal hardware that I have.

[vapid lava]

Well, I have a bunch of varied hardware I can test it on if you want

There's one PC with a 7th gen intel CPU and a GTX 1060 3GB, there's one with I think a 3rd or 4th gen intel CPU and iGPU, then There's of course my main PC, with 12th gen intel and an RTX 4070 Ti, then there's that one laptop that has a, 8th gen intel and iGPU

And those are just the ones I'm pretty sure are 64 bit

I think I might have one more 64 bit system with a Pentium of some kind, but I'm not 100% sure about it.

[halcyon prism]

Well, if you borrow x86 firewood, then you can start creating met, I’m not sure

[olive rivet]

This would be my second attempt at doing this. I previously made a prototype operating system just to see if I could even develop one. I learned a lot during that first attempt.

[polar apex]

i've got a Laptop with a Ryzen 7800, RTX 3070 Laptop, and Radeon Graphics
if for any reason you'd want to test a laptop gpu seperate to a desktop gpu

[olive rivet]

I agree: it will be hard. I'm doing it primarily to challenge myself and learn new things.

[halcyon prism]

and as a result?

[olive rivet]

I'm assuming that you're asking what happened to it. I ended up scrapping the prototype around 6 months ago; it wasn't intended to become anything serious.

[halcyon prism]

It’s a pity

I hope it works this time

[olive rivet]

I don't think so. That would be a massive undertaking.

[olive rivet]

I'll definitely take a look at that for inspiration.

[vapid lava]

Ok, I think that would be a good window style. A windows compatibility layer would have been cool, but you can leave it out if you don't think you can implement it. Maybe you can return to the idea if you feel like it

[olive rivet]

It might be able to gain a Windows compatibility layer if I can get Wine to run through my POSIX compatibility layer:

https://www.winehq.org/

[vapid lava]

I know about the Wine Windows compatibility layer for Linux. That could technically work

[halcyon prism]

I think that this could work only with Unix core systems

[vapid lava]

Just make a Linux compatibility layer to run the Windows compatibility layer on

[polar apex]

"OSs are like Onions: They have Layers"

https://tenor.com/view/shrek-onions-have-layers-layers-onions-gif-25090894

[halcyon prism]

Limits in real live living chat

[vapid lava]

When you make your own OS, the only real limitation is your CPU

[halcyon prism]

And fantasy

[polar apex]

and your skill

[vapid lava]

Skills can be improved though

Improving the CPU architecture would be a whole another level of complexity

[halcyon prism]

It’s another task

[vapid lava]

Changing the CPU architecture to a custom one would require testing and manufacturing CPUs, which is quite a bit outside of the skillset of most people

[halcyon prism]

A bit, yeah?

[vapid lava]

So if I understood correctly, the GUI, once added, is going to be mouse and window based like in most modern OSs?

[halcyon prism]

We’re talking about Linux?

[vapid lava]

I was asking about this OS's GUI

[halcyon prism]

Yes, it will work as other graphic OS

But because we’re using Linux as a base we’ll have big opportunity to modify it

[vapid lava]

But this OS isn't Linux based

[halcyon prism]

Oh, then you can build whatever you want

[vapid lava]

But assuming it will have a mouse interface, the cursor could probably use the existing Neuro themed cursor

[halcyon prism]

You have to write it

[vapid lava]

It would need to be written to work, but I think the Neuro themed cursor would look good

And work well with this OS's goal of being Neuro themed

[halcyon prism]

I think that if we can made whatever we want, we can built OS controls by the ai, no cursor at all(joke)

[halcyon prism]

Yeah, I think that this really gotta be good looking

[vapid lava]

Do you know what the existing Neuro themed cursor looks like?

[halcyon prism]

Mhm

For example vedal

[vapid lava]

Do you want the zip? It can be used on Windows

[halcyon prism]

?

[vapid lava]

You know I'm talking about something someone on this server has already made?

[halcyon prism]

I think we can photoshop some art to make it

[vapid lava]

It exists already

[halcyon prism]

Wow

It’s ok, in my opinion

[polar apex]

imo, it's a bit busy

too much neuro, too little actual cursor, so to say

[vapid lava]

Well, it fits the theme

[halcyon prism]

This is fan OS

What do you know about arg?

[vapid lava]

Not much

[polar apex]

but a fan os should still feel nice
the easiest thing would be including the option to swap cursors (similarly to windows)

[vapid lava]

That would work

[halcyon prism]

We can include something for these

[halcyon prism]

Of course

It is obvious

[silver ruin]

interesting project im curious how far you will manage to go with it

[formal hull]

I can’t wait to use it!

[halcyon prism]

I think we can include an AI for some features in OS

[vapid lava]

Including an AI in an OS would greatly increase the system requirements, since AI is very computationally expensive

[halcyon prism]

And what about server’s AI

[silver ruin]

guys we don't even have anything past booting into the text and you chat about AIs XD

[vapid lava]

If you were running the AI on a server, that would mean you need a server. And servers are expensive, although I could let people from the Neuro community host their Neuro stuff on my PC, but an AI (assuming it's an LLM) might be a bit much even for my PC

[halcyon prism]

Yeah, now we’re just thinking of plan for the future)

[halcyon prism]

On git there is a model of OS and there’s servers

[silver ruin]

well it really depend what this AI would need to do and there is always an option to ship 2 versions one with local AI othet without AI

[halcyon prism]

I think he is thinking of LLM

[silver ruin]

if he does then ye you need decent pc to run LLM

at least locally

[halcyon prism]

Yeah it’s really challenging

[silver ruin]

nothing like seeing 100% CPU usage on btop while running it :V

[polar apex]

he has a 4090 for a reason and that's cyberpunk 2077

[halcyon prism]

:;)

[vapid lava]

And I only have a 4070 Ti. My CPU is technically better than Vedal's though, assuming he still uses that i9-10900K

An i7-12700K is better

[halcyon prism]

Is it a laptop or pc?

[vapid lava]

It's a desktop

[silver ruin]

yk that we talking about overall requirement for this OS not if we could run it or not :V

[halcyon prism]

Wow, pretty cool

I have only combined 2x 3060ti

[polar apex]

i have a 3070 laptop (unthrottled)

[vapid lava]

My PC is a bit overbuilt for most applications. I'm planning on upgrading it to 64GB of RAM and maybe trying out some LLM stuff on it, maybe training one off of Evil and combining it with my Evil TTS

[silver ruin]

damn im just running 3060

[polar apex]

dang, 64 gigs base ram?

i have 32 gigs base + 16 gigs sharable vram

[halcyon prism]

What is your relationship with higher mathematics?

[vapid lava]

The 16GB sharable VRAM is just half of your RAM, it's not additional physuical memory

[polar apex]

then it's not vram

[vapid lava]

Math is fun. I like doing it

[polar apex]

i was told the card has it's own phys ram

and i'd be getting 48 gigs of ram in total, just with 16 gigs being part of the gpu

[vapid lava]

Video cards do have their own VRAM, for a 3070 it's 8GB

[halcyon prism]

Pretty cool to have this stuff and be interested in mathematics

[polar apex]

guess i was lied to

[vapid lava]

Another highlight feature of my PC is probably the fact that it has 25TB of storage

[polar apex]

wtf

why

[vapid lava]

Why not.

[polar apex]

that's server level of storage

[silver ruin]

dude just create NAS lol

[halcyon prism]

Do you use this often ?

[polar apex]

what, the card?

i do, i like to play demanding racing games for example

[vapid lava]

Ignore the D: drive, it's a backup

[polar apex]

oh i hate the fact that this PC has partitions

windows and partitions doesn't work

[halcyon prism]

What are you doing on your pc?

[polar apex]

and it's screwing me over regularely

[vapid lava]

Literally anything I want. Like once I trained a TTS off Evil's voice

[halcyon prism]

What type of rom it is?

And you’re using only 32 gigs of ram

[vapid lava]

Do you mean RAM? ROM is uncommon in new computers

[halcyon prism]

How?

[halcyon prism]

I mean rom, stationary memory

[vapid lava]

ROM means Read Only Memory

You might mean what kind of drives they are

[halcyon prism]

I mean stationary, I don’t remember the abbreviation

[vapid lava]

The BOOT and PROGRAMS drives are SSDs and the 2 DATA drives are HDDs

[halcyon prism]

good luck filling them out in the next 2 years

[vapid lava]

For more specifics on what I do on my PC, I make Neuro mods for games, help test Neuro related computer projects (like the Neurolings) and do some video editing. And sometimes AI

I also just store whatever random stuff happens to be interesting. Including everything that has Neuro in it, I probably have 100s of Gigabytes of Neuro related things just sitting around

But I guess you could call it a bit of a server. That's what I've started calling it sometimes

[halcyon prism]

You’re right

[olive rivet]

I made some good progress today on the serial console driver and shell. Here's a short video showcasing that.

[vapid lava]

Now you just need files and commands and you have a DOS

[halcyon prism]

Good progress 👍🏻

[polar apex]

[olive rivet]

I made significant progress within the past few days. In particular, I was able to implement and enhance the following functionalities:

Heap Memory Allocation

To date, there hasn't been a way to dynamically allocate memory at runtime. I had been relying on stack memory for static allocations. This became a significant hurdle when I was trying to write a shell parser: I needed to be able to parse input of unknown sizes. To address this issue, I ported and implemented a linked-list allocator into my memory module. This means that I am now able to use the Rust allocation and collection libraries in my operating system.

Shell Commands

Last time, I demonstrated that I was able to enter input into a console shell prompt. However, there weren't any commands that could be run. To address this, I implemented two basic commands:

echo - Displays a line of text to the terminal.
help - Prints the command list.

If the user enters an unknown command, an error appears in the output below.

Memory Detection

The memory module is now able to parse the memory map that is provided by the bootloader. This is currently being used to find usable memory for the heap allocator. When it finds a region, it prints a debug message using the logging system and gives the memory to the allocator.

CPU Detection

There is now a basic symmetric multiprocessing (SMP) module that is able to find the amount of cores that a processor has. Currently, it prints a debug message to the screen using the logger. This will be used in the future to implement multi-threading capabilities into the operating system.

Serial Console

The serial console now has the core::fmt::Write trait implemented. This means that it can now be used for writing and formatting Unicode buffers and streams.

Bug Fixes

There were issues with the way that the console shell handled backspaces and carriage return characters. In particular, it was possible to erase the entire shell prompt line. These issues are now fixed.

[formal hull]

This looks awesome!

[vapid lava]

Are you going to implement a static set of commands, or have it like in Windows where executables the system has a special pointer to the path of are also commands in the command line?

[olive rivet]

It's most likely going to be like UNIX/Linux, where there'll be a directory called /bin that contains the executable binaries. Those binaries would be available to the shell for execution.

[vapid lava]

Will you be able to define additional paths for that, or is it going to be hardcoded?

[olive rivet]

I'm aiming to have it not be hard coded. To do that, I'll need to add environment variable support for the shell. That way, I can define an environment variable called PATH. Users will be able to add additional paths to that variable.

[vapid lava]

Sounds like how it's done in Windows

What commands are you planning to have by defaut?

[olive rivet]

In the long term, I'm aiming to implement or port the set of commands that are defined in the POSIX standard. The list is here:

https://en.wikipedia.org/wiki/List_of_POSIX_commands

In the short term, I want to implement a couple of commands for power management (e.g. shutting down, rebooting).

[orchid solar]

Woah. I have tried my hand at some OS dev in the past, but I burned out really quick for probably a few different reasons. Either way, I'd love to take a look at this, and possibly contribute

What type of learning materials are you using? Just the OSDev wiki or are you referencing any textbooks?

[olive rivet]

I've looked at the following resources for guidance:

https://wiki.osdev.org/
https://www.reddit.com/r/osdev/
https://pdos.csail.mit.edu/6.828/2023/xv6.html
https://pdos.csail.mit.edu/6.828/2023/xv6/book-riscv-rev3.pdf
https://wiki.minix3.org/doku.php?id=www:documentation:start
https://os.phil-opp.com/

I plan on writing contributor guidelines at some point. I'm not sure when I'll get around to it.

[orchid solar]

Yeah, I'd definitely have to do some learning before trying to contribute. I still haven't gotten to learning Rust for real, and I'll need to brush up on my limited OS dev knowledge and, of course, expand it

[olive rivet]

I've definitely enjoyed my time learning Rust. It's refreshing not having to deal with footguns that are found in languages like C/C++.

[orchid solar]

Yeah, I've been pretty interested in sitting down and learning it for real. Out of curiosity, though: what's your level of programming experience?

[olive rivet]

I have ten years of programming experience. I started to learn how to program when I was 17.

[orchid solar]

Ah, welp.

I'm 22, and I've been programming since... I dunno, middle school? But I haven't actually programmed for real much. I feel like I have a lot of theoretical knowledge, but little experience. Anyway, I was asking for reference just to see how I should tackle this

[olive rivet]

Honestly, I don't think the years of experience matter too much. It's more about the motivation to learn and your willpower to keep going in the face of obstacles. OS development doesn't provide much in terms of instant gratification; it can take a while for your project to blossom.

It certainly took a bit of trial and error before things started clicking.

[orchid solar]

Yeah, uh. I don't know why I thought I should've made my own bootloader and so I stuck in 16-bit real mode and ended up making a display driver using the memory-mapped video display stuff...

And uhh. Not much payoff, honestly

[olive rivet]

I haven't tried to develop a bootloader yet, but I imagine that it is a gigantic pain.

[orchid solar]

This was probably a few years ago

[orchid solar]

It is... You have to set up your own interrupt table and shit just do upgrade to 32-bit, and you have to add more to get to 64-bit

At least, if I remember correctly

[olive rivet]

For reference, I'm using this bootloader for my operating system:

https://github.com/limine-bootloader/limine

It gets you straight to long mode (64-bit), and it's pretty easy to configure and build your operating system against. That's in comparison to something like GRUB.

[orchid solar]

Oh god, this was when I was 15

[vapid lava]

When will you want to test this on real hardware?

[olive rivet]

It can probably be tested at any time, but I'd like to get a VGA terminal/console, ACPI driver, and PS/2 keyboard driver implemented first. That way you'd be able to administrate the system without needing a serial port connection.

[vapid lava]

A serial port really isn't something I have

But what about USB peripherals? I don't have PS/2 peripherals either, at least confirmed working ones

[olive rivet]

I think USB keyboards can fallback to using PS/2 if a driver isn't available. I'd need to confirm that though.

[orchid solar]

Are you just targetting UEFI?

[olive rivet]

I'm targeting both BIOS and UEFI.

[vapid lava]

Isn't that fallback BIOS or UEFI level, dependent on if the motherboard supports it?

[orchid solar]

Whoops, I thought that you could access UEFI drivers after the boot process. Apparently not

[lilac trail]

Well, this will be interesting. Will you be doing dev streams?

[olive rivet]

I'm not sure yet. I'd have to give it some more thought.

[lilac trail]

All right.

[vapid lava]

I don't think a fallback like that would be part of the keyboard, but I've heard of some devices like older Macs that could boot Windows having it

[orchid solar]

Ugh. Apparently you have to write 4 drivers to fully support USB

[vapid lava]

Even if that fallback was a feature in some motherboards, it's most likely not a keyboard built in fallback and probably not supported on every motherboard

The USB hardware is not PS/2 based, so having the fallback on the keyboard doesn't sound feasible to me

[orchid solar]

We're talking about at the BIOS level

Probably

[vapid lava]

Is there a PS/2 compatibility layer built-in in the BIOS?

[orchid solar]

Ah, I lied. But there is PS/2 emulation in the pipeline (in some firmware, but I can't find where), though it can be kinda sketch

[vapid lava]

I think I'd prefer a proper USB keyboard driver rather than a somewhat functional emulated PS/2 interface

[orchid solar]

That is true, but it's in the middle of being developed and they're doing it to learn. Let them work on it how they want and for however long they want

[vapid lava]

Yeah, whatever works I guess. By the way, there are apparently plans to make a window-based interface for the OS later, so do you think the Neurolings could run on it?

[orchid solar]

Let's keep that to the other thread

[vapid lava]

Ok.

[halcyon prism]

Are you going to do only system console commands or it will be a type of uefi commands too?

[vapid lava]

#1200735329104171068 message These are the commands that are apparently planned to be implemented

[olive rivet]

I'm currently planning on implementing commands just for the system.

[halcyon prism]

I see

[vapid lava]

Are you gonna implement a filesystem or more commands first?

[olive rivet]

I'm most likely going to continue implementing more commands. I've already added a command today called logs which retrieves the system logs.

[vapid lava]

Ok, but you will need a filesystem before file manipulation commands can be added

[olive rivet]

That's correct. I'm leaving those for later.

[vapid lava]

Ok, will be interesting to see how that works

[olive rivet]

It's been a couple of days since the last status update. I've attached a video that demonstrates some of the newly implemented features. Here are the areas that I've been working on:

VGA/Framebuffer Driver

Like I did for the serial console driver, I've implemented the core::fmt::Write trait. This means that the driver can now write or format Unicode buffers and streams. It also gives me access to the write!() and writeln!() Rust macros, which I've been using extensively for outputting text to the screen.

Due to this change, I also needed to come up with an abstraction for setting the coordinates or colors of objects before I draw them to the screen. This is especially important when using the above macros (they do not expose any coordinate or color options). I ended up creating a struct called Cursor which contains a set of coordinates, a foreground color, and a background color. In a way, it sort of acts like a paint brush.

There is now also a clear() function that wipes the framebuffer canvas with the color black. This will be used in the future for transitioning from the introduction screen to the VGA terminal.

Logging System

The logging system has been completely revamped. Instead of directly writing logs to the screen, it stores them in a vector data structure that is held by the Logger object. Each log is a struct that contains a log level and a message. They also implement the core::fmt::Display trait, which means that they are formatted correctly when written to a buffer or stream.

Console Shell

On the backend, the console shell has been refactored to be an object. There are also a few new commands that have been added:

logs - Retrieves the system logs from the logging system.
time - Displays the elapsed time.
pwd - Prints the name of the current working directory (this is currently hard coded due to the OS lacking a file system).

The prompt line has also been changed to mimic a UNIX-like system.

Serial Driver

The driver has been refactored to use the Port type from the x86_64 library that I've been using. Previously, it had directly used inline assembly to read and write from a serial port.

Programmable Interrupt Controller (PIC)

I've been working on a driver that manipulates and queries the 8259 PIC. This is used for controlling the CPU's interrupt mechanism, which in turn will allow me to retrieve timer, keyboard, and mouse events. Currently it masks/disables everything except the timer interrupt.

Timer Module

I have made a module that implements a basic timer. Whenever a timer interrupt is handled, the elapsed time that is stored within the timer is incremented by one. This is currently being used to support features such as the time command and text fade-in.

[vapid lava]

Seems like progress. Will it support real hardware with a USB keyboard soon? I'm interested in testing it once it does

[olive rivet]

That is the plan. Once I implement the keyboard driver, I'm going to be testing my wireless USB keyboard on a bare metal install and see if it works.

[vapid lava]

Sounds good. Let me know when it works and I'll test a bunch of various hardware

[twilit stirrup]

I've been looking at this progress, glad to see it is doing well! Great job honestly.

[olive rivet]

Thanks, I appreciate it. I initially thought it wasn't going to get this far; I'm glad I was proven wrong.

[olive rivet]

I've added support for mounting an initrd, also known as an initial ramdisk. This allows me to load a temporary root filesystem into memory as part of the startup process. With this implemented, I was able to decouple resources such as fonts and images from the kernel itself (making it smaller in size). Processes now need to retrieve their resources by loading them from a filesystem path. The attached video showcases the files and directories I was able to load from the disk. Each trace log message shows the file headers and data (as individual bytes).

[vapid lava]

So it loads files now?

[olive rivet]

Yep.

[vapid lava]

Is it capable of saving files, or will that be implemented later?

[polar apex]

does this mean i could load my own font?

[olive rivet]

Nope. The ramdisk filesystem is read-only for the purposes of bootstrapping the system. Writable filesystems such as FAT and Ext2 will be implemented later.

[vapid lava]

That's what I expected

[olive rivet]

Yes. However, you would need to generate a new initial ramdisk that contains the font.

[olive rivet]

Here's another status update of what I've been working on (I've attached a video to demonstrate some of the features below):

Process Model

I've been trying to model what a process in the operating system looks like. Currently, each process has an identification number (ID) and a state (running or stopped). It will be later expanded to include a context (the set of data that must be saved to allow a process to be interrupted and later continued from the same point).

Process Scheduler

In order to support multitasking, I needed to implement a basic scheduler that queues processes in a fair manner. I ended up writing a basic round-robin scheduler that assigns time slices (quantums) to each process. Each process is currently assigned a time slice of 100 ticks (which is intentionally slow for testing purposes). When the time slice for a process elapses, it is put into a stopped state and moved to the back of the queue. The next process in the queue is moved to the front and put into a running state. This continues on until the operating system is shut down. In addition, there are trace messages that get logged whenever these transitions occur.

Keyboard Driver

I've been working on a very basic PS/2 keyboard driver. It is able to read scan codes that are received from its data port (0x60). There is also an interrupt handler that causes the keyboard to interpret said scan codes. Currently, only the ENTER key has any value or significance to the operating system. Whenever that key is pressed, it clears the screen and spawns a VGA terminal. And finally, every scan code received is logged as a trace message.

VGA Terminal

To support systems without a serial port, I've been working on a VGA representation of a terminal. There were a couple of changes that needed to be made in order for this to work. For one, I made the VGA driver have the ability to interpret escape sequences (including ANSI). Second, I generalized the shell display function to accept anything that implements the core::fmt::Write trait. This allows me to interchangeably use my VGA and serial implementations for that trait. Lastly, I removed the singleton object that I made for the shell (this was needed so that multiple shell instances can coexist). At the moment, I am working on generalizing the shell interpret function so that it can work in either a VGA or serial console environment without any issues.

Logging

Files that are found in the initial ramdisk now have custom formatting when logged. The log message tries to mimic the output of the tar -tvf command to the best of its ability (given the current constraints). There is also a new log level called Fatal that is used to designate unrecoverable errors (such as kernel panics). The panic handler for the kernel has been updated to take advantage of this new log level.

[vapid lava]

How many bits are used for the task ID?

[olive rivet]

It uses 32 bits. It's stored in a u32 variable.

[vapid lava]

Is it unsigned? (I'm not 100% sure what that variable type is)

[olive rivet]

Yep, it's unsigned.

[vapid lava]

Ok, 2^32 processes isn't that bad for a cap, I don't think I've ever seen anything even close to that many

[olive rivet]

I'd be amazed if it could support that many.

[vapid lava]

Most users will probably only at most have a couple hundred or a thousand processes, and that might be a high estimate, but who knows, some more advanced users can definitely get a lot more

That is based on Windows process numbers though, not sure how many this OS will have

[olive rivet]

It's hard to say. I think it's going to be less than a hundred (at least to start with). It might increase significantly once a desktop is added.

[vapid lava]

The process count increasing a lot when the desktop is added sounds very likely. The desktop allows for a very good multitasking environment, allowing for multiple programs to be open and on-screen at once

[olive rivet]

I'm finally able to render the Neurolings after writing a parser for the PPM (Portable Pixel Map) image format. I still have some cleanup to do, but I figured I'd give a sneak peek of what I've been working on.

[vapid lava]

Wow, that's some rendering right there. Will you be trying to port them to the OS?

[olive rivet]

The program? The usage of Java unfortunately complicates that. If it had a native port (e.g. Rust) then it might be feasible.

[vapid lava]

DeathByPrograms is making a rust port

[olive rivet]

@open cloak Do I have permission to use your Neuroling sprites for this project? I want to make sure that it's okay before I commit them to my GitHub repository. You will be credited within the README file.

[open cloak]

yep sure!

[olive rivet]

Thanks.

[vapid lava]

You should probably keep an eye on the Neurolings project to see when the Rust port is finished, if you think that would make it possible to port

[olive rivet]

I definitely will. I'd be happy to help them port it once I make substantial headway with the desktop.

[vapid lava]

Cool, I think that would be a really cool feature to have

I do think every system should be able to run the Neurolings.

[orchid solar]

The Rust port isn't a certainty, though

Also, the libraries I'm using wouldn't support a different windowing system unless quite a bit of work was put. And I don't know how wgpu would work with it

[olive rivet]

It's been over a week since my last status update. Here's what I've been working on:

Grayscale/Color Images

The image renderer has been enhanced with support for grayscale and color images. This is possible because I wrote a parser for the Portable Gray Map (PGM) and Portable Pixel Map (PPM) image formats. The renderer initially only had support for Portable Bit Map (PBM) images, which were very limited in versatility. This feature is demonstrated on the introduction screen, where there are now colored sprites of Neuro and Evil.

CI/CD Pipeline

I added a CI/CD pipeline to the project. It is driven by GitHub Actions, and it runs after a commit is pushed to the repository. Currently it only builds the ISO image by running the make all command. I am planning to enhance this pipeline with jobs that target different development platforms (e.g. Windows and macOS).

Documentation

The README has been enhanced with new information for debugging the operating system with GDB. The architecture diagram was also slightly tweaked to reflect that servers are the middlemen between user applications and drivers. Finally, the prerequisite list has been updated to show the current dependencies that are needed for developing the project.

Process Context

I added a context structure to each process. This is needed to support my preemptive scheduler. In the future, this will allow each process to be interrupted and resumed at a later point by saving the state of each CPU register.

System Configuration

I have been adding system configuration files within the initial ramdisk. Within the initrd/etc/ directory, there are now three files:

hostname - Stores the current hostname of the system. This is set to localhost by default.
motd - Stores the message of the day (MOTD).
passwd - Stores the users that are on the system. The root user (also known as the superuser) is the only one defined at the moment.

These files are used when instantiating the console shell. In particular, the passwd and hostname files are used for constructing the shell prompt line. On the other hand, the motd file is used for displaying information before the shell prompt appears.

Executable Parsing

There is now a module that can parse ELF (Executable and Linkable Format) files. In addition, there is now a readelf command that can print the file header of an executable. This module is needed so that we can launch user programs in the future. To test this feature, I created an executable program called init that currently does nothing.

As always, you can view the project source code here: https://github.com/Theomund/NeurOS.

[vapid lava]

Once you properly implement users, will you add a password system too?

[olive rivet]

Yes. Just like on Linux, there will be a command called passwd that will allow you set the current user's password. Behind the scenes, that password will be securely hashed and stored within a file called shadow within the system's configuration directory.

[vapid lava]

Sounds like a decent system

Will you handle most of the OS functions in the kernel or external executables?

[olive rivet]

The idea will be to move as much functionality as I can to userspace using external executables. That way I can fulfill my goal of writing a microkernel.

[vapid lava]

Ok. I think that might be how Windows works, but I'm not sure. At least it has a lot of processes of it's own

[olive rivet]

Progress is picking up again. I had to halt development for a couple of weeks because I didn't have any free time (my job got in the way). This week's main focus will be on getting preemptive multitasking to work.

There's also a roadmap (WIP) that's now in the README file:

https://github.com/Theomund/NeurOS#roadmap

[vapid lava]

Ok cool

I was wondering where you disappeared to

[olive rivet]

I've been making a considerable amount of progress for the past few days. Here are some of the highlights:

Bold Font

I have added a bold variant of the Terminus font into the operating system. It's currently being used to display bold characters within the VGA terminal.

Documentation

There is now a roadmap section within the project's README file. It delineates the current areas of focus for development. Please keep in mind that it isn't an exhaustive list, and it will be expanded over time as things start to unravel. I also tweaked the overview section so that it's more apparent of what the project's goals are. Finally, there is a screenshot section that showcases the operating system as it currently stands.

CPU Exceptions

I have added interrupt handlers for all possible CPU exceptions. There are 23 handlers in total, and most of them currently just log the exception and move on. I will try to make them more comprehensive in the future.

Logo Image

I changed the logo image so that it's a Portable Pixel Map (PPM) image. It used to be a Portable Bit Map (PBM) image, which caused it look pixelated. It should now look a lot cleaner when it's displayed in the introduction screen.

Global Descriptor Table

One of the most fundamental components of an operating system is the global descriptor table, which is used to define memory segments and access permissions for those segments. In the past, I have simply just used the table that is provided by the bootloader. As time went on, I realized that I needed to define my own so that I can add my own segments. With that in mind, I added a custom global descriptor table with segments for kernel code, kernel data, user code, user data, and task state. It is the first thing that gets loaded when the kernel initializes.

Physical Memory Manager

There is now a physical memory manager that parses the memory map for usable regions of memory. When it finds a region, it breaks it up into consecutive blocks of physical memory called frames. Each frame is 4 KiB in size, which matches the size of a page (a block of virtual memory). The addresses of those frames are stored within a list.

Virtual Memory Manager

There is also a virtual memory manager which manages the page table, a data structure that stores mappings between virtual addresses and physical addresses. On startup, it grabs the active page table from the Cr3 control register. When the kernel wants to allocate memory, it has to provide a range of pages that it wants. The manager then maps each page to a physical frame, which is allocated and provided by the physical memory manager.

Heap Allocator

The kernel's heap allocator has been tweaked to request memory from the virtual memory manager. During the initialization process, it requests for 4 MiB of memory starting from the 0x444444440000 address.

Here are the images of the updated logo and the new bold font.

[vapid lava]

Seems like you're making decent progress

[wide mural]

what the heck

screw bsd I'm using this

[junior fossil]

here
https://puter.com/
https://github.com/HeyPuter/puter

[umbral pollen]

This is awesome

[olive rivet]

Thank you all for the compliments.

[olive rivet]

I saw this on Reddit a couple of months ago. The idea of an operating system running in a browser is a pretty cool concept.

[olive rivet]

I'm currently working on adding a user space to the operating system, which is where normal applications will reside in with reduced privileges (Ring 3). In the meantime, I improved the kernel panic handler so that it clears the screen and prints all system logs that led up to the panic.

[vapid lava]

Alright, seems like you're making progress

Are you gonna have a JVM in the OS?

Since that's what Kotlin compiles to, so I could develop software for the OS

[olive rivet]

I'd have to see how complex the port would be. I'm initially aiming to support Rust and C within the operating system. Other languages, if they're not complicated to port, will gain support later.

[vapid lava]

Well, C and Rust are compiled languages, which don't compile to a specific runtime, while a JVM is a runtime environment for Java and Java compatibles

So C and Rust just go onto the CPU, while JVM is a layer that sits in between the CPU and program

[olive rivet]

Yep, I know that. If I recall correctly, the JVM is primarily written in C++. I believe I'd have to port over a C++ standard library implementation first.

The one I'll be porting over for C is this: https://github.com/managarm/mlibc.

[vapid lava]

Well, if you manage to get JVM support (specifically Java 21 JVM), let me know and I can maybe try seeing if Kotlin runs on it

[olive rivet]

Would Python be a viable alternative? I think that might be easier to port from my end.

[vapid lava]

I don't think I can do Python, I'm primarily a Kotlin developer

Python is also ran differently, which might actually make it harder to port. Python is also slower, since it's ran as plaintext instead of compiled

I do recommend JVM, since that has more games written to it too, so technically the OS could run Minecraft

Python is really not a common language for games

[onyx heath]

I think JVM impl is very far away 😄 The GPU Drivers, which are required to run those software

[olive rivet]

Yeah, it's definitely far out (a year or two). There's a lot of stuff that still needs to be done before it becomes a priority or even feasible.

I'll definitely put it on my backlog though.

[vapid lava]

Alright then, will be interesting to see where this OS project will go

By the way, the latest version of the Neurolings runs on JVM 21

So you'd get support for that by implementing all the stuff it needs to run the JVM

Although if the Rust port gets done at some point, that'd also work

[olive rivet]

After doing a bit of research, it looks like Lua is going to be relatively easy to port to the OS. That might be the third language that gets supported after Rust and C.

[vapid lava]

Well, whatever works. Just keep in mind I'm waiting for JVM support to start developing for the OS

[olive rivet]

Development is resuming again. I'm hoping that I'll have something new to show by this weekend.

[half geyser]

You have unlocked new role

[vapid lava]

Alright, cool

[fossil hedge]

what was this? and was it ever moved to #1336777692070023271 ? (I'm writing a list of old projects)

[twilit stirrup]

creator is no longer with the swarm

checking the user proves that