Crystal Realms - 2d sandbox mmo game

[eternal turtle]

Hi everyone! I am working on a 2d sandbox mmo game called Crystal Realms. This game has been in development for the past 3 years, but the last update I have switched over to using bevy as the game engine.

The server is also written in rust and has been using the bevy_ecs crate for the past 2 years.

The game already has many features working:

• Crafting
• Quests
• Achievements
• Potions
• Parkour mechanics
• PVP
• Monsters
• Trading

The pre-alpha version of the game is available on https://crystalrealmsgame.com or https://marcomeijer.itch.io/crystalrealms and we already have a small community of active players.

We are also planning to release the android version of this game before the end of this year to the google play store.

[stoic aurora]

the last update I have switched over to using bevy as the game engine.
What did you use before bevy and how was the switch? 👀

[eternal turtle]

I used libgdx before

The switch went smoothly, there were a few bugs/missing features of bevy that affected my game. But most of those are fixed now, or will be fixed in 0.15.

[abstract arch]

Looks sick!
What do you use for networking?

[eternal turtle]

I used to just use plain tcp, but now I am using renet https://github.com/lucaspoffo/renet

[abstract arch]

Looks like all big projects roll their own high level networking solutions on top of renet 😅

[eternal turtle]

Oh yeah, I also have my own networking stuff built on top of renet. I am not even using bevy_renet but just the regular renet.

[vernal pewter]

do you use anything that syncs the world like replicon or do you send "raw" packets?
i.e. specific packets for specific things like "PlaceTile", "MoveEntitiy" etc.

[eternal turtle]

Idk how replicon works, but I am basically doing it with raw packets. And I have a special system in place to route the messages to one-shot systems like this.

[vernal pewter]

Ah I see, nice!

Replicon basically syncs all entities that have a special Marker component for you.

[stoic aurora]

Renet everywhere, no one can escape the renet. bevy_replicon used to be coupled to it, bevy_bundlication started coupled to it, and before bevy_replicon and lightyear were in a good spot almost everyone used it 😂

What does the networking look like other than sending messages trough a layer on top of renet? Is there anything like client prediction involved?

[eternal turtle]

Yep there is client prediction. For anything that the client has to predict I basically create a command that implements this trait:

pub trait UndoableCommand: Send + Sync {
    fn execute(&mut self, world: &mut World);

    fn write(&mut self, world: &mut World) -> Message;

    fn undo(&mut self, world: &mut World);

    fn read(&mut self, world: &mut World, stream: &mut dyn Read);
}

The write is to send it to the server, and the read is to see if the server accepted it, or what the server thinks should have happened.

Then all of these commands are added in a command queue. And before processing the messages I receive from the server, I undo all the commands I executed locally, to make sure the server and client execute the things in the same order.

[stoic aurora]

Oh, that's a rather unusual approach but it actually seems a lot less complex to set up and debug than a rollback system, with the drawback being that you need to implement new things manually, but the upside being that it scales by how often things happen rather than by things predicted * tickrate

[eternal turtle]

Oh I thought this was called a rollback system. But basically I implemented what is explained in this article: https://gabrielgambetta.com/client-side-prediction-server-reconciliation.html (And also what is explained in part III, but I didn't do part IV because it is not really relevant to this game)

[stoic aurora]

Usually when people talk about a rollback system it refers to a system that records state to a history, then when new data comes in, it loads data from that history and reruns every frame that happened since

[eternal turtle]

Yeah thats what I am doing. These commands get added to a history (Well I called it command queue). And whenever new data comes in from the server, I undo everything. And after I process that data I execute everything again.

[stoic aurora]

In the case of recording the state you just have the benefit of not needing to undo anything, at the cost of keeping a lot of histories

[eternal turtle]

Ahh I see, I think in my case that would have been more expensive/complex.

[stoic aurora]

And things going catastrophically wrong over the tiniest bug, my favorite rollback bug is this one I created recently: #1211045598854127636 message

[eternal turtle]

Haha

[stoic aurora]

Yea, I can definitely see that. I've been working on a full rollback system for quite a while now and it has been a massive pain. Before this I did what many shooters do, which is basically just take a shortcut by rolling back to a full server snapshot for the player, and only keeping inputs since then + histories for things that aren't predicted. That removes most of the edgecases that can be created, but only predicting the local player turned out to be far too limiting to make the combat I want (action combat that heavily relies on having a bunch of interesting and unique skills)

And besides just making the system being a pain, I also had to optimize my game's simulation so it can run say 14 times per frame

[versed berry]

This looks awesome! How are you storing the world/grid data?

[eternal turtle]

In a global resource, this resource just has an array of all the blocks (each block is stored as an u16)

[versed berry]

Chunks or anything like that?

[eternal turtle]

Chunks are not really needed in this case since worlds are not infinite. Each world is 300x170 blocks (which is more then enough for this type of game, since players often want to fill a whole world with their creations)

[versed berry]

That's great to hear, actually. I am also working with a grid for my game and worry too much about optimizations before they are necessary. You only see one 30x17 sized chunk at a time and I was thinking I was going to need to load them in and out.

[eternal turtle]

I just released a new trailer for the game: https://www.youtube.com/watch?v=numZYr2iAqE

[abstract arch]

Looks very cool!

I wondering what is your experience with Bevy. Do you you the latest version or stick to specific one?
Curious because recently discovered that some big projects don't migrate.

[eternal turtle]

I am using the latest version of bevy. Since I am targeting mobile devices it is really important for me to have all of the latest bug fixes on mobile (and I also make use of some of the newer features in 0.15)

On the server side however I am using an older version of bevy: 0.12, but on the server side I only use the bevy_ecs crate.

[abstract arch]

Interesting, thanks!

[stoic aurora]

How was the 0.15 migration? Did you get hit hard by bundles -> required components?

[eternal turtle]

Oh it was a pain. I think it took me like 14 hours to fully migrate everything? But honestly I think some other things were harder like the changes made to the rendering, because I have a few custom rendering pipelines. The first time I got the game to compile again almost everything stopped rendering or looked completely wrong.

[eternal turtle]

Crystal realms just got published to the google play store 🎉 https://play.google.com/store/apps/details?id=com.crystalrealmsgame.crystalrealms&hl=en

[eternal turtle]

The game has really been blowing up in indonesia this weekend. Today we reached 200 players online.

[stoic aurora]

Is this a new record? If yes what was the previous peak?

[eternal turtle]

Its a new record by a LOT. The previous record was last week when the game launched on google play with 38 players online.

[stoic aurora]

Nice, how are the servers holding up, or is the game structured in a "players run their own instances" way? 🤔

[eternal turtle]

The servers are holding up pretty well, before the launch I was running the server on a cheap VPS for 5 euros a month. And after stress testing I was able to figure out it was able to handle at least 50 players online.

For the launch I upgraded to a dedicated server for 100 euros a month that is many times faster in case the game got a sudden spike in players like this.

I even forgot to increase the thread count on the server, and even with just two threads it was able to handle 180 players. (the server has 48 threads)

[stoic aurora]

Nice ... So far I've also been hosting my game (which is still in a pre-alpha state so basically no players) on a 5 euro VPS 😂

With the whole two cores I have I can handle between 30 and 100 players, would probably be a mistake to release a demo without getting a more legit server 😂

[eternal turtle]

Yeah maybe

[eternal turtle]

Also you might need to keep in mind if your server provider has ddos protection. My previous one didn't, and one guy kept ddos attacking the server asking to be unbanned and demanding crypto.

[stoic aurora]

That's a good point actually ... Also having things like bandwidth caps could easily turn into a way to take a server down

I think the host I'm at (hetzner) should provide DDoS protection, tho I'm not sure how good it is

[eternal turtle]

Update: today we even reached 680 online at the same time

[abstract arch]

How servers are holding up?

[eternal turtle]

Today is the first time the server started to lag at 740 players

[stoic aurora]

I guess you're the biggest bevy game now, ahead of even Tiny Glade 😂

[eternal turtle]

It was close a few times before, but today the game reached 1000+ players online at the same time!

The past few days I have been fully focused on optimizing the servers, fixing bugs, adding profiling tools and that kind of stuff to handle this increase in players.

[stoic aurora]

So the servers aren't on fire with 1000 players? 👀

[eternal turtle]

Nope, after optimizing it, it runs pretty smoothly. My main bottleneck is that I was using one mutex a lot, so a lot of threads spend time idling waiting for it to be unlocked. Another optimization I did was running the server on multiple udp ports. This way I was able to split the code that was responsible for sending networking packets over multiple threads. And when the players first connect I redirect them to one of these ports. (Thanks to renet this was very easy to do, and I didn't even need to update the client)

Sometimes there are lag spikes in popular worlds though when it gets autosaved. The servers cpu usage is at about 70% with this many players, so I definitely need to upgrade the server if I want to allow even more players.

[stoic aurora]

Huh, I thought UDP is already supposed to be thread save without multiple ports 🤔
Does the server run as one big app with many different instances inside it?

[eternal turtle]

I am not sure, maybe it would have been possible. But I don't know how I would have done that using renet. I am basically running 4 instances of renet on my server, and I am not sure if it is possible to run one instance of renet on multiple threads. This also fixed the issue that renet has a limit of 1000 connections. So by running 4 instances I can support up to 4000 players on a single server

[stoic aurora]

Ah of course, not everyone is using a weird custom transport with renet where they can do whatever ... I've been doing that since I switched to renet so it's the only approach I know 😅

[stoic aurora]

Hit any other milestones after the 1000+? 🤔

[eternal turtle]

The current record of number of players online is 1800+

But that was about 2 weeks ago, and right after the game got DDoS attacked for a few days which made the playerbase shrink a bit. But right now we still reach a peak of 1300+ players online daily.

I am gonna release an update soon, so maybe after that we will hit a new record.

Oh and the discord server hit 10k members today! That was a pretty big milestone

[gloomy canyon]

wow! congrats! 🎉

[abstract arch]

I wonder who these guys who DDoS indie projects 🤔

But I guess it's also means that the game is quite popular, congrats!

[eternal turtle]

It was a competitor kinda

They host an illegal private server of a similar game as mine, and they probably saw me as competition.

[abstract arch]

Illegal? They reverse-engineered someone else's game?

[eternal turtle]

Yeah, they rewrote the server code of growtopia (a game by ubisoft). Someone told me they host the biggest private server of that game.

[abstract arch]

I wonder why reverse-engineering games is fine, but when it comes to servers it's illegal 🤔

[eternal turtle]

Actually I am not sure about the legality of it, but I am guessing it would be copyright infringement or something similar since they are making money of it, and the real game is still being run by ubisoft.

[abstract arch]

Ah, makes sense!

[stoic aurora]

In a lot of cases it's actually down to how the game gets distributed ... Reverse-engineering a game is fine, making a patch for the game to make it connect to your server is fine, but releasing a custom client for your server is not fine

[stoic aurora]

That's honestly kinda sad

[lyric garden]

Any higher traffic? Max players / core?

[lyric garden]

How many 48 core bare metal servers for 1300 players? Players / core? I see this is only Google Play store, not Apple store. Did apple not approve?

[lyric garden]

Maybe cloudflare for DDOS protection?

[eternal turtle]

I switched to a different server provider that has better DDoS protection. This server has 12 cores and is able to handle about 3000 players it seems (each single core on this server is faster compared to a core of the old server that had 48 cores)

[eternal turtle]

Apple hasn't approved the game yet, they are being very annoying in the reviews. I got denied a few times because they said there was no option to delete the account, even though there is one in the settings. And I gave them instructions on how to do it.

[eternal turtle]

And about that, as far as I understand cloudflare mostly offers tcp protection. Their udp protection is a bit more pricy.

But my new server provider uses gcore for its DDoS protection. And I was able to protect myself against a 870 Gigabit/second DDoS attack without the players noticing anything.

[lyric garden]

Which provider is this superior gcore UDP DDOS protection? I didn't know cloudflare requires an upsell for UDP.

[eternal turtle]

Vibegames, I got recommended it from a moderator of the game. I am sure there might be better providers though. But this one works well with our game.

Also because I changed the networking library of the game from renet to enet. And vibegames has specialized DDoS protection for enet.

[eternal turtle]

Its been a while since I posted here, but I have recently released a very big update:

[lone ferry]

are you planning a release on steam?

[eternal turtle]

Yeah, when the official release is gonna happen. Probably gonna set up a steam page in the new few months so we can start getting wishlists