Rewriting the game in Rust

[junior hollow]

The game is quite good and i love it, but i think that the choice of writing it in lua (or any other interpreted language) is a horrible one. The game should have been written in a fast, secure, resource efficient language like Rust, or at worst C++. So i think it would be amazing if the developers started rewriting the game in Rust as a side project.

[iron lodge]

It's already C++/lua

[junior hollow]

I see the game's code here https://github.com/beyond-all-reason/Beyond-All-Reason and i think 93% of it is in Lua, am i missing something ?

[iron lodge]

You're missing this part https://github.com/beyond-all-reason/spring

[junior hollow]

This is the engine not the game, could it be that the engine compiles the lua code to C++ ?

[iron lodge]

BAR is using this engine

[cosmic fiber]

spring does a vast majority of the heavy lifting

there's lots of widgets, gadgets, other logic, automated tasks, unit defs, etc built into the lua on BAR repo - I doubt any of that is doing nearly as much work as the engine itself...

[junior hollow]

Build Back Better

[obsidian hill]

It’s been a while since I checked, but the lua scripts can take anywhere up to about 20% of CPU (probably more when Lua AIs are involved).

Part of the immense value of lua scripting, which currently significantly outweighs the performance cost, is lua appears significantly easier to reach proficiency in.

So while using C++/rust/whichever may bring a significant performance improvement, performance is currently generally good enough that we’d rather just have the higher dev capacity that Lua provides.

[junior hollow]

According to this research

https://greenlab.di.uminho.pt/wp-content/uploads/2017/09/paperSLE.pdf

Lua is one of the worst languages when it comes to memory usage, energy usage and performance.

I believe that programers should not just write software that works, instead they should make sure that the program is as much conservative as possible when it comes to resources and has the best possible performance.

Rust is as highly productive as any other interpreted language you find out there, plus it's one of the most secure language available , plus it's as fast as C++ or even better.

No wonder it has been the most loved language among developers according to stack overflow, for 7 years.

I see no reason to use any language other than Rust in writing software in 2022.

I think that developers should avoid using interpreted languages unless they really have to, like Inside the web browser for example.

Maybe rewriting the game fully in rust will give it a nice performance boost.

[iron lodge]

"I see no reason to use any language other than Rust in writing software in 2022."

1. Not everyone know Rust (who gonna write code?)
2. With Lua no need compile code everytime, with Rust you need.
3. Legacy issues.

[still epoch]

lua appears significantly easier to reach proficiency in.
had basic familiarity with java, read some lua tutorials, tried to understand custom Spring widgets, got absolutely filtered. is it really that simple?

[iron lodge]

Plus rewriting entire game will take really long time

[safe heart]

@junior hollow "I think it would be amazing if the developers started rewriting the game in Rust as a side project."

This whole game is already a side project xd

It's 10y+ project

[junior hollow]

Well Rust is really worth learning, so I think every developer should learn it, along with C and C++ for studying existing code.

Cargo the build system and package manager for Rust saves you the pain of compiling the entire code everytime. It's something like make but much much better.

[junior hollow]

Yea that maybe the case, but I think it will be worth it. A new operating system is being written in rust now, called Redox. Tor had been rewritten entirely in rust. Cloudflare had rewrote their proxy server in rust, instead of nginx. Parts of Linux kernel will be replaced in rust ... etc

[junior hollow]

I think Java is a nightmare. Look at android studio and it's performance.

[uneven pagoda]

I will just drop a line that ends all discussions like this.

Pull Requests welcome!

[obsidian hill]

Nice try :)

[obsidian hill]

I kind of enjoyed working in Java, to be honest. Having to write types everywhere made some things easier to read, and changed the way I write Swift code.

[obsidian hill]

Yes.
Maybe.
I did write “appears” rather than “is”, the bigger point was trying to emphasise our top priority.
The biggest thing with Lua is it’s a small language. For someone working full time, the time to learn all language mechanics in rust is probably negligible, granted. For Spring, where many people will write one 50 LOC widget in their life?

If I’m wrong, it would be awesome to have that demonstrated. Again, I care more about advocating for our selection criteria than what we’ve selected.

[junior hollow]

There are still many computers out there with 4GB of ram and 2 core CPU, if you run vscode which will take nearly 1GB of RAM along with Firefox which will be the same, you would have all your RAM busy and your PC will start getting slow.

What I wanna say is that if you have a full OS with all it's apps written in Java or python or Lua, you may be able to run maximum 4 apps let's say at the same time. On the other hand, having the same OS with all it's apps written in Rust or C++, you will be able to run 10 apps or even more.

The language used affects the end user greatly both in CPU/RAM usage and energy consumption. So I see that even if we spend some more time to develop an app just to make it more efficient in the end, it will be worth it and is the right thing to do, cuz you have to prioritize the end user experience.

When I was working in web development using vuejs, I was running, Firefox, vscode, postman and they were using my whole goddamn 8GB ram. Thanks to javascript.

[obsidian hill]

Indeed, I think the same way too. I personally own a raspberry pi and it's a little disappointing Spring doesn't run on it – I know this feels facetious because there's no way it would ever run on something with that little power, but it's just what comes to mind.

Web apps bother me immensely for similar reasons you've outlined here, just to further establish that "I understand".

Moreover, there is and has been significant effort to increase the performance of the engine and the game, and it's worth noting that Lua's footprint is generally at most 10% – 20% in the worst cases. Sure, this is huge compared to some things, but it's not the difference between unplayable and playable.

What's also worth noting is that if we didn't write in Lua, we would likely have nothing at all. And again, by all means, demonstrate me wrong, but based on my conversations with a lot of the other BAR devs, Lua has been enough of a challenge. I don't have experience with rust, so I can't measure its approachability vs Lua. I'm just saying, that's the bar you'd have to cross.

I don't know enough, but I'd imagine there's more space for rewriting parts of the engine in rust than the game itself. And at times, I think even parts of the game have been moved into the engine's C++ when it's clear that a performance improvement can be made there without compromising the engine's ability to serve multiple games.

An alternative effort which might actually be orders of magnitude easier would be moving to JIT-compiled lua, which has been discussed many times, had implementation attempted at least once, and would possibly bring the total overhead of Lua down to about 1-2%. depending on the benchmark (quick google produced this: https://github.com/kostya/benchmarks) we'd still have 1/5 the speed/memory efficiency of rust, but we'd still have devs.

I've actually dug into the engine's Lua API enough to determine that you almost definitely could (somewhat easily, afaict) add a layer to load up a dylib written in, well, anything that can output a C-compatible dylib. ||This is not something I've done before, so I might not know what I'm talking about.|| The problem is more, most of the people here just wouldn't be able to use it. Unless they could – in which case, let people know, and maybe we can make the switch.

But the bottom line is that people here are making the game better in whatever way they can – and as far as I know, rust just isn't something that a lot of people can currently do, or learn in a reasonable amount of time compared to their contribution.

This isn't supposed to be a knock on anyone: I can't write rust. I can't even write very much C++, and I'm bottlenecked on current projects by my inability to effectively learn how the engine does stuff.

So yes, they're gonna keep making the game better, but in ways they can.

And maybe one day, if there's capacity for it, a rust (or other language) rewrite will likely be in order.

[junior hollow]

Yes Rust is little bit hard to learn cuz it has some new concepts that does not exist in other languages. Those concepts are corrections and improvements of bad buggy designs is C++, Rust is considered the new C++ ("That's what it's designer said"). Once you learn Rust, you will have the ability to use one language for nearly everything. Rust can be used in low level system development such as Operating systems, drivers and embedded with performance similar to C/C++, and it can work perfectly in high level user-space applications due to it high level capabilities. Also guaranteed memory safety is why Rust had become so loved and popular. Not to mention the complete official tool-chain that comes with Rust.

I am currently working on a GUI library in Rust (That uses the Material Desgin) that will be used to build a desktop environment and all the apps in Redox (A new OS written entirely in Rust). When i am done, maybe i will spare some time working on spring in Rust from scratch. The game is good, i loved it and i think it deserves to be written in Rust.

[obsidian hill]

Wow, nice

Idk how the engine deva feel about rust but they might be interesting to talk to about rust in the engine. They might be able to geek out with you more. But alas, my expertise is all in Swift :D

[daring saffron]

I find it hard to believe that either of the following will be useful:

1. Talking about rewriting the entire engine in rust.
2. Someone coming in and quietly rewriting the entire engine in rust over the course of a few years.
   I think a useful rust port would have to happen rapidly, over a few months, and by an expert in spring (ie someone who knows "the requirements" of the engine, what is merely an implementation detail and what is important, what can be optimised, what has to be maintained).
   If someone can manage that, and existing devs turn out to be fine with it at the end of the few months of rewrite, then it sounds good to me.

The fact of the matter is that Spring works. You can't really argue it doesn't - games run on it fine. We would have to be in a place where Spring doesn't actually work to argue existing devs into scrapping everything to work on a rust version. No amount of saying "the foundation of the project is bad, here is how I would want it done better" gets around that, because the project works in spite of any comment from the sidelines.

Not to discourage you from joining engine dev, learning the deep spring knowledge required, and porting it all to rust in a few months at some point. I've fiddled around with rust a bit and I like its paradigms. If I could snap my fingers and have the engine (this includes all social and knowledge infrastructure around it) seamlessly ported to rust then, I'm not sure I would do it right away, but I would certainly look a bit deeper into rust to check for potential issues, since rust seems promising.

[junior hollow]

I love to write software that not just works fine, but the best software I could ever write. Sometimes is can be a boring and lengthy task to rewrite something in Rust, but the results are amazing in terms or security, performance and future development.
https://blog.cloudflare.com/how-we-built-pingora-the-proxy-that-connects-cloudflare-to-the-internet/
https://blog.torproject.org/announcing-arti/
https://www.rust-lang.org/static/pdfs/Rust-Tilde-Whitepaper.pdfhttps://www.rust-lang.org/static/pdfs/Rust-Tilde-Whitepaper.pdf

[daring saffron]

it would also help if you talked less like a press release

[cosmic fiber]

Remember the opportunity cost... you can't just argue that Spring/BAR should be rewritten in Rust just because "Rust is good." Even multi-billion dollar companies with vast amounts of resources don't operate on this logic, let alone collaborating devs of an open-source project. There has to be enough reason to commit to this change over doing something else with that time. That also includes taking into account other potential cons of the change, like those mentioned before of requiring developers to learn the new language.

security, performance and future development.
In the blog you posted, Tor received a grant from Zcash, and the privacy focus with Tor necessitates very high security, performance, and development ease to keep up with its large user base, value, network requirements, and new protocols. Cloudflare is also a very large company whose application is basically sold on its security and performance.

Performance is a factor for BAR, but "good enough" is good enough especially if it's able to run on lower-end systems (there's a level of performance investment expected for video games in general, whether BAR wants to push those boundaries is up to the devs). There are many other things that could be a higher priority for the devs before performance becomes a pressing issue: new features, campaign, tutorial/new user experience, bugfixes, UI updates, units, maps...

I believe that programers should not just write software that works, instead they should make sure that the program is as much conservative as possible when it comes to resources and has the best possible performance.
Not to mention the fact that the goalpost for "best" moves all the time as technology and software evolves.

I love to write software that not just works fine, but the best software I could ever write.
As noble as these sentiments are, perfect takes time. We can't let perfect stand in the way of good enough.

[obsidian hill]

There are many other things that could be a higher priority for the devs before performance becomes a pressing issue
Actually, priority is a highly pressing issue for a certain set of developers – particularly Ivand, TarnishedKnight, Floris, Beherith, afaict. They actually have relevant skills in that area – Ivand/Behe in graphics, Floris/Beherith in Lua, TarnishedKnight in architecture & multi-threading. Others, like IceXuick (who isn't really a dev tbh) have a much greater emphasis on UX because that's where they're skilled. The pattern is, the devs use what skills they already have to make the game better.

the goalpost for "best" moves all the time
Kind of. The "best possible", maybe. But the reason the goalposts move is because they have a set of static goalposts: "Does infinite good in 0 time consuming 0 power."

So it sorta makes sense, if Keikai is highly skilled in rust, performance, and code quality, that his effort would be using rust to improve the game's performance and code, and that sounds awesome!

Maybe the one thing I'd recommend is not trying to make every part of the body an ear. Having a wide range of abilities allows greater resilience, as we can solve a greater range of problems. That said, if we can upgrade to better versions of ourselves, why not?

[cosmic fiber]

That's a good point you bring up that each dev has different skills and focus areas. Then the opportunity cost narrows to those few devs working in that particular niche, and rejecting a volunteer solely because their skills don't align with existing development patterns is unwise and a cost in itself.

[obsidian hill]

rejecting a volunteer solely because their skills don't align with existing development patterns is unwise and a cost in itself.
Exactly, though on the other side of the coin, it can seem very difficult to work out where you (or someone else) can fit in. There are many professional devs in the community who don't contribute – some of which aren't interested, because their day job is enough for them, and some who just don't have any clue where they'd fit in. If we could find someone skilled in fixing the latter problem, that would be awesome, but it seems like the Spring community in general has been pretty bad at converting potential devs into actual devs. (It seems way better now, though!)

[gleaming vessel]

Why not Zig, you can compile C++ with Zig compiler and implement only the new stuff in Zig language 😄

[obsidian hill]

Ooh that sounds interesting. Any other cool facts about Zig?

[gleaming vessel]

You don't have to wrap C/ C++ libraries to be able to use them, you can just include the header files

You can cross compile to any platform on any platform

[obsidian hill]

That’s not the new language the SerenityOS peeps did? Wait, I remember, that was Jakt

Well ok thnx google

[junior hollow]

Some benchmarks:
https://github.com/kostya/benchmarks

[safe heart]

@junior hollow are you rust expert and familiar with C++? You could try demo rust integration in engine by taking some small component, rewrite in rust and do interop with existing code base using something like https://cxx.rs

E.g. this https://github.com/beyond-all-reason/spring/tree/BAR105/rts/System/Config This module needs to be mostly rewritten anyway as it's buggy. (we get file corruption sometimes)

One Dev started rewriting https://github.com/beyond-all-reason/pr-downloader/ in rust: https://github.com/gajop/sprd, you could contribute there, and try to integrate that in engine build instead of pr-downloader.

And for pr-downloader I've even written permissively licensed functional tests that verify the end binary behavior https://github.com/beyond-all-reason/pr-downloader/blob/master/test/functional_test.py so that's a help when rewriting.

[junior hollow]

Yes i know Rust, C++, C and some Java. I have also worked for a while in web dev using JS and php. I know about CXX and i don't think it's good to use something like that in any new project if you are aiming for correctness and clean code. Currently, i never use C/C++ except to study and do some modifications in existing C/C++ code.

I think that eventually everything will be rewritten in Rust. There have been many bad conventions and habits in the software world and most developers now see those habits as ok, cuz these habits have been in place for years. Even the most experienced engineers never complained about those bad conventions. Those bad conventions and habits showed their problems throughout the years.

One of these problems is, the unnecessary use of interpreted languages and relying on them in lot in many fields where they should not be used, like desktop apps, web server etc.

Another problem is buggy unsafe languages like C and C++ have been there for years and people still use them despite proven countless times that they may result in catastrophic security bugs, some of them can stay hidden for years without being discovered.

So i think that the most correct and reliable solution is to rewrite from scratch with a language like Rust that solves all the design problems and bugs.

In the upcoming years you will find many projects being rewritten in Rust. This will also help greatly not only in security and speed, but also in the ease of future development.

I mention Rust simply cuz its the best candidate i see now. I have used Go for a while and it has problems like garbage collector that prevents it from being used in low level dev

In the end this is my personal opinion. I will do my best to improve the Rust ecosystem such as libraries to make it easier to rewrite things like Spring.

It will be cool to see the existing version of BAR and another version in Rust and comparing them to see which one performs better

[safe heart]

You don't need to spend any more time on advertising Rust. That's not the point.

Rewrites of big pieces of software often fail and are very risky. Rewrite of spring in rust from scratch is not going to happen. Period. Unless suddenly there are a buch of developers who can spend a lot of time doing just that. You are heavily underestimating the size of the project.

A lot of projects are applying rewriting component by component. See Firefox usage of Rust, inclusion or Rust in Linux kernel. All those are incremental.

[junior hollow]

It will be incremental of course, but as a separate project. There is nothing wrong about writing a project beside the existing one from scratch. It will not be like a full time job writing lets say spring from scratch

Operating systems are being rewritten, so what prevents spring form having that

Many developers are against the idea of rewriting everything in Rust i know. I am just trying to share my ideas here, not force everyone to use Rust

[safe heart]

It will be incremental of course, but as a separate project
That then it's not incremental, that's rewrite from scratch.
Operating systems are being rewritten, so what prevents spring form having that
This https://github.com/beyond-all-reason/spring/graphs/contributors

That's why incremental approach can have some merit, and rewrite from scratch doesn't. Spring was being written by a few people over the course of 10 years+, non incremental rewrite would take multiple years again. At this point you might just screw spring engine altogether and try to migrate game (so also entirely rewrite from scratch) to some other existing modern one

And rewriting means simply rewriting, not much actual features, improvements, just rewritting, for a long time.

I was above suggesting attacking small existing components, rewriting them, e.g. pr-downloader, you didn't respond to that. Your response is "all touched by C++ is doomed and have to be purged and rewritten from scratch", well, that won't happen.

[junior hollow]

That's what i said in the title of this suggestion, "Rewriting the game is Rust".

I don't think the project is that much big that it will take "years" to rewrite, the project "spring" is nearly half million (523_550) lines of code, while a project like vscode is a nearly a million and half (1_433_169) lines.

I think if a team of 10 Rust developers studied Spring's code well, they will be able to rewrite it in a few months.

[safe heart]

I think if a team of 10 Rust developers studied Spring's code well, they will be able to rewrite it in a few months

Maybe?

But good luck finding 10 rust developers being able to spend close to full time on this open source project 🙂

[junior hollow]

Let's see where things will go

[safe heart]

I'm also not getting why you are bringing up vscode? Written in TypeScript with many engineers in a paid team at Microsoft

[junior hollow]

I am just comparing here project size

But to be honest, there are things more important to be written in Rust rn that i think a game will come after a while. We are still at the GUI library point here in Rust. Desktop environment, operating system etc

[daring saffron]

things will only go if you take them somewhere