Architectural question - real-time game server processing

[bronze walrus]

Hi, everyone!

I'm in the process of building a MMO RTS game (yes, a very niche subgenre!). I have some general architecture questions of how to handle some more complex use-cases, but in particular - game state.

You see, players queue up actions of things they would like to do to manage their empire. This can be things like:

• Construct an infrastructure project
• Send a fleet to attack a player
• Send cargo craft to trade between planets
• Research technologies

.etc.

I originally started writing this in PHP before realising that the real-time requirement on the server-side was just going to be too difficult to handle/solve. So what do I mean by "real-time" in this post?

Let's say for example that a player starts constructing some infrastructure. While that building is being constructed, their planet is attacked and their available resources is diminished to a point where they can no longer sustain all the activities on their planet (let's assume they're researching some technology whilst also constructing a building). The logic here, is that the system will use a FIFO approach, whereby - each item being produced/created, starting from the end, will be paused until there are enough resources to continue constructing the remaining items, at which point, once those are completed, the next item in the list would then be resumed.

Similarly, let's say that a player has those same items in progress, and they complete some research that improves construction time for any buildings. At that point, I would then need to update those constructon projects by reducing their total time to completion.

Where this gets tricky (and why it's quite challenging (although not imkpossible) in PHP with say, backend queue servers), is that when these events occur, the system needs to recognise it's impact across the game's state. In this instance, I would need to then update those running processes and disable the right ones, whilst also doing things like notifying the player, updating the UI.etc.

I'm curious how you would best solve this using Elixir. My view here is that I'd have all these long-running processes (some could take hours, or even days), and somehow manage them by sending them messages to pause/cancel/stop.etc.

[sly raft]

Managing state for a game like this, for me, would look like an Ecto database where each player has a set of flags to say simply if the construction is built or not. Depending on that flag, handle the game logic accordingly. If you want it all handled server-side, it'll be cheap to handle the load of 1 player with 200 buildings to process. 50 players, might become a little more spicy. 500? Hmmmmm. 5000000 and you're gonna have to organize a queue of some kind. I believe there is a system that exists that lets you effectively execute Elixir Cron jobs- So for when you initialize a construction, take the initial supplies (20 wood, 5 stone), post to DB that the player has spent that wood and stone, then push back to that cron process with a 'construction built time'. On complete, execute that building's 'complete' status.

I'm working on Rayex right now, which is more grounded in the layer of desktop right now so anything I offer you is purely hypothetical at the moment, but I am aware it can be done because I've thought about networking quite a lot. I would look into Agents and try to find the package I can't remember the name of, because it gives you a lot of power to schedule something. I suppose the main idea is how much of this do you want to process on Elixir. Are you shifting the whole thing to Elixir? If that's the case, your new requirements will change. But I believe setting up an elixir cron job to postback the PHP would be sufficient, and Agents can manage your state.

[oak crescent]

@bronze walrus how do you plan to organize these “games”? Are there potentially many of them going on at the same time? How many players in each

[bronze walrus]

@sly raft the database would manage the overall state, but that's not what I'm referring to - or asking about. The fact is that these processes need to be happening in real-time, so that they can respond to events happening throughout the global game state - in real time 🙂 So although postgres will act as a SSOT, it's not the only state that has to be managed. I literally need to the ms updates to processes (I think)

[bronze walrus]

How long is a piece of string? 😛

Ideally I would like to see 10s of 1000s of players in each galaxy. Each player I would think has quite a few things going on. The objective is that at endgame players have between 10-15 planets (any more becomes too difficult to manage), and so each planet would have its own tasks going on, as well as all the tasks for fleets, trade routes.etc.

If I presume that there would be (I still need to dive deeper into this) at endgame, 15-20 tasks per planet, and they have the same in total # of planets, and they've got another 50 or so trade routes, fleet commands.etc. going on - we're looking at yes - upwards of 500 of processes, per player, per galaxy.

However, as to my other thread regarding multiple servers - there is likely to be 1-2 galaxies per server.

As I undersatnd it however, Elixir could manage those processes relatively easily - 500 per player, at 10,000 players - is 5,000,000 concurrent processes. If I'm deploying to an "elixir cluster", that should handle that fine, no?

[oak crescent]

It sounds like a lot of processes to me but I don’t really understand what you’re building. It’s more difficult to synchronize things across processes so I think that should be a main concern. If you need certain things to happen in the same sequence together you’ll want that happening in one process if you can do it that way. You can always spawn off tasks to execute stuff concurrently to your program continuing to operate too

[bronze walrus]

PS. I'm quite confident of attracting that many players - the niche genre I'm building in is old, stale - it needs to be reinvigorated, and there's a lot of players looking for something a "bit more special"

[bronze walrus]

Yeah, I don't think that can happen - they're generally very different processes/requirements.

Is it possible to fire events that other processes can listen for? That's one way of doing it.

[bronze walrus]

Think a real-time strategy game, but massively-multiplayer. Take everyting a single player does in a usual RTS game (all the things going on in the background) - now apply that at scale, and you should get a feel for the challenge(s) I'm facing. hehe

This also is going on even while players are offline.

Btw, I'm totally open to other ideas here - my thinking is likely wrong, as I'm still yet to do a proper dive into Elixir.

[oak crescent]

It sounds fun and not that niche. RTS is having a resurgence

[bronze walrus]

Yeah but that's a different kind of RTS. haha

MMO RTS is very niche

[oak crescent]

Is this gonna be like one huge map or what?

[bronze walrus]

BTW, one of the reasons why I lent more towards Elixir, is that if I can have these processes running, and have them react to global game state as required, it's much, MUCH faster than doing an action, updating the database, firing a domain event, having those side-effects managed, that may also fire domain events.etc.

[bronze walrus]

Yup! Galaxies are broken up into sectors, systems and planets.

But you may be thinking about it differently to what the game actually is - it's mostly text based with a lot of imagery to spice it up - think games more like Endless Space.etc.

[oak crescent]

What’s the scale of a planet? Like how many folks taking actions on a single planet

[bronze walrus]

Here, this might be a better starting point for you: https://halcyon-online.com/halcyon-online-a-primer/

It's a space game - players manager their empires, which includes planets, fleets, trade, asteroids.etc. So 1 player per planet, and they will have many.

[oak crescent]

Do all the planets only belong to a single player?

[bronze walrus]

Sorry, I don't quite understand your question. Players manage a space-based empire - they'll be colonising planets and managing them.

[oak crescent]

Could a planet be controlled by more than one player?

[bronze walrus]

No.

Players can absolutely lose their colonies though, in more hostile sectors.

[oak crescent]

Maybe it makes sense to have all the planets running in one process per player.

[bronze walrus]

Hmmm... that's an interesting idea. Perhaps I've been thinking about processes all wrong 🤔

How exactly would you see that work?

Each player would have an "Empire" process of their own?

What about tasks that are completed or interrupted by other events?

[oak crescent]

It’s a common thing for folks new to beam to use more processes than necessary

[oak crescent]

Potentially yes. Maybe spawned when they connect?

[oak crescent]

Can you give an example

[bronze walrus]

That won't work, because things have to be running while they're offline 🙂

[bronze walrus]

Using the example above - they're constructing a project, perhaps they're hit by a pirate raiding party, which takes some of their resources for a time - I have to then go through the in-progress projects on the affected planet, and pause projects until their available resources are able to cover the projects being worked on. Does that make sense?

So, an outside event, that is relevant to the empire, affects a single planet, and its associated projects.

Similarly, a research project is completed which reduces construction time - I'd need to then go through every single construction project and update their relevant completion times, to the second.

This is why just using a database falls over - it can't really manage this particularly well.

I could potentially use something like Redis tbh, with a TTL on objects, and then update the TTL, which could work - but that has other problems like if the side-effects fail for some reason - that data in Redis is gone and recovery to a prior working state is impossible.

In many cases, there are many things affected as actions are completed by the player, and they all need to happen as quickly as possible, with an ability to recover if any of those side-effects fail.

Eg. If the construction project is interrupted by a raiding party taking resources - the taking of resources, and the pausing of relevant projects needs to happen as close to simultaneously as possible. Think database transactions (which also don't work in this situation hehe)

The way I was viewing it previously, was like this:

1. Raiding party is successful, and takes x resources
2. The relevant processes are notified -somehow-
3. Relevant processes then update their state, time to completion.etc.

This is why I was viewing it in a very granular manner.

Does that all make sense, @oak crescent ?

[oak crescent]

I think so but gotta go afk for a bit. Definitely an interesting problem. A lot of the timing stuff seems like it could be precalculated or use absolute date/times instead of timers when the user isn’t connected. Like if a construction project gets hit just modify the completion timestamp accordingly. Run it through the client player process to handle publishing to the client if they are connected otherwise just see when they connect what projects completed_at dates are in the past. That sort of thing

[pulsar oasis]

Or just use elixir :) here's a nice blog post by josé about redis and elixir: https://dashbit.co/blog/you-may-not-need-redis-with-elixir

[bronze walrus]

Yeah, that's how it's working currently in PHP and Laravel with backend queue servers, but when there's a lot to "be done" due to an event, the delay can be a few seconds as it goes through database, processes data, executes code, fires other events.etc. Queues though become hard to manage, as I then need to let a command that's in the queue when it has to run to complete the action.etc.

This is all (I think) managed much more easily with processes that can speak to one another when required to do so.

[bronze walrus]

I wasn't actually suggesting Redis, due to what I said 😛

[bronze walrus]

That said, pub/sub is very close to what I need - but I need that feature set with redundancy and retries.

It may very well be that I need to speak to other servers in the cluster if the game is successful 🙂

Hmmmm, yeah.... might be what I need... @pulsar oasis - do you have any further insights regarding redundancy and failure recovery when using phoenix pub/sub?

[bronze walrus]

Was thinking about the # of processes again... Correct me if I'm wrong here, but the less granular the process is in its function, the more state i'll need to manage. Here's an example - if I have a process for each separate action that takes time to complete, that process is doing just one thing - ie. complete construction, land fleet, conclude combat. If for example, I instead have say, planet processes, each planet process needs to be aware of much more than is going on at any one time...

The way I was thinking of doing it, was spinning up a process that just sits idle while it waits for its timer to complete. However, whilst doing so - it can then be interacted with directly. Ie. When it spins up, it could then subscribe to any events that would affect it, and react as required.

Aka, although there may be a fuckload of processes running, they're doing almost nothing 99.9% of the time.

[oak crescent]

Yeah you would be managing more state, but that makes management easier because you’re not having to coordinate everything across processes

[bronze walrus]

Issue is if a planet process fails due to bad state, it'll be more difficult to recover from, no?

[oak crescent]

There is a way to hibernate processes if you want to go that route. Processes are a limited resource and do consume memory so that’s worth considering though the biggest concern I have is managing the state and that can become difficult across processes

[bronze walrus]

I don't need to manage state across processes, they'll manage their own. I just need to ensure they listen to and react to the appropriate messages 🙂

I guess I can always refactor into a planet process if it becomes unwieldy.

[oak crescent]

Well it sounds like they’re coordinating with each other and that can get tricky if you can avoid it it’s nice

You could just log out the results somewhere stable like Redis or use something native to OTP. So if there’s failure reload from that

If you crash multiple times maybe keep another copy to try instead or have some way to recover from known good state.

[bronze walrus]

Yeah, they have to. But it's not as though they're telling each other what to do, rather just notifying them of global state changes.

"this might be of interest to you"

"Cheers!"

[bronze walrus]

Can you elaborate on this point a little so I better understand?

I think pub/sub is the way to go - because I don't want to have to directly tell processes they have to do something when an event occurs, I'd rather they react to global events -if- it happens to effect them.

And I think I can do that so that when the process starts (ie. BuildInfrastructure), that it subs to the appropriate events it knows it would be affected by.

[oak crescent]

It’s easier working with one big data structure handling one event at a time than having processes talk back and forth, potentially deadlock, etc with weird bugs that only show up in production use.

If you’re using Phoenix you could potentially use Phoenix pubsub. I’ve used gproc_ps before and had success but that was long ago and there may be better options now. You could also use an external message bus too

Like rabbitmq, some cloud service, etc

[bronze walrus]

Deadlocking processes?! O.o

How would that happen?

Oh, you mean like if one process dispatches an event that a sibling reacts to, which then dispatches an event that the original process would be interested in?

[oak crescent]

Basically any concurrency bug can happen when you move to using concurrency. Race conditions that can turn into deadlocks. Just logic bugs in your code where you didn’t consider receiving events in every different permutation of order they arrived in

[bronze walrus]

Ah I see.

Doesn't that make the argument for the smallest possible processes @oak crescent ?

[oak crescent]

I don’t see how it does. The more processes you have the more that can go unexpectedly

[bronze walrus]

What I mean is, let's say each process is roughly 4-5 lines long - isn't that better than say, having a Planet process that is potentially hundreds of lines long?

To be fair, I know such a process would likely utilise many modules, so the code isn't directly in the process itself.

[oak crescent]

No I don’t think that really helps. You can still structure your program to respond to different events with minimal data passed down in the same way state and code could be split across processes.

[bronze walrus]

Could you elaborate a little on that? I guess I should start diving in and figuring this out rather than just pestering you 😛 Just trying to do a lot of planning/homework first before getting to the truly challenging stuff.

[proud wyvern]

Thought from left field? Can you not store an initial state, then calculate a new state at any future time when you need to change the vector? Ie- you know that you’re collecting resources at rate x/hour, if the user logs off, you store the rate and wait for some trigger, and free up the resources. Pirate raiding party happens, you update the conditions to time raided, handle the result, then reset the conditions until the next relevant event.

[bronze walrus]

I'm some cases I can, but not all. It becomes unwieldy pretty quickly - that's how I was doing it originally in PHP. It's also considerably slower when there are other side effects to manage at the same time.

[bronze walrus]

What I mean by my above point @proud wyvern (if I understand your suggestion correctly), is that having to depend on the database, and then have side-effects be managed whilst also dealing with the database, adds considerable latency if lots of things need to be updated at once by different processes. This is much better and faster if I can just manage it in the application memory itself, and then store data as required (as updates to global state are necessary).

[quartz dust]

https://www.theerlangelist.com/article/spawn_or_not relevant article about the steps to decide whether to spin off a new process or not

[bronze walrus]

Oh amazing, ty!

@oak crescent I found this, too: https://www.youtube.com/watch?v=WjKQQAFwrR4&t=1367s&ab_channel=ThePrimeTime

At the end they talk about how long it took to spawn up processes and handle them at 1 million - in nearly all cases, 12s to complete lol. In my case, that is certainly not acceptable.

So your advice to manage processes in some sort of grouping, is apt 🙂

@quartz dust oh BOOM: Do not use processes (not even agents) to separate thought concerns.

As that's how I was thinking about them haha

[quartz dust]

I’m developing a MUD server in elixir (an mmo server with accessed via a TUI). As for processes, each connection, character, room, and zone is a process.

[bronze walrus]

Holy shit, I haven't heard of MUDs in YEARS!

@quartz dust I'm building a space-based MMO RTS, the processes I felt were suitable were:

1. Tasks that have to execute at a particular point and
2. That's it. lol

Even planets could be massive, considering each galaxy can have 10s of 1000s of planets.

[quartz dust]

Might want to do each planet as a process and if there is too much activity split them into sub zones

[bronze walrus]

Perhaps I need to lean more into how I was doing it with PHP and Laravel, but instead of queue servers (delays), just use processes?

[quartz dust]

As for dead locks, you don’t want like processes to be able to call each other — to use an MMO example: you don’t want two characters to be able to call each other because if character A calls character B while B is processing some code to call character A, then they’ll time out waiting for a response from each other

[bronze walrus]

What about using an event system?

[quartz dust]

Event based works great with elixir

[bronze walrus]

Using elixir over a queue system would be vastly more performant... But I would still need to manage things to cancel them.etc. Ie. Let's say I have a project to complete in 3 hours, but something causes that to be paused, or cancelled - I'd need to update the data in postgres, and then potentially stop a scheduled task.

[quartz dust]

Pattern matching on event topics is nice

[bronze walrus]

Wait what!?!? 😮

[stiff marlin]

If those benchmarks are what I think those are, those are very flawed https://hauleth.dev/post/beam-process-memory-usage/

[bronze walrus]

@quartz dust how would you advise handling the use-case above?

[stiff marlin]

Tldr, it's much closer to Java virtual threads

[quartz dust]

If I’m reading correctly, since all projects on a planet are to be pause—able all at once, it sounds like it might be useful for each planet to be a process. Planets could have an event handler to handle events like pausing projects or cancelling projects

[bronze walrus]

I'm wondering if it's needed, though? Each project could be its own process (it kinda has to, as it has to execute at a particular time) - but can that scheduled task be set to listen to events and cancel itself or update state, before it executes? If not, I guess Planet is the right process to manage.

It's just that there can be a lot going on with a single planet.

That said, the # of planet processes would be considerably smaller than the # of action/project processes, which would cover everything from fleet travel, combat resolution, construction projects, trade routes.etc.

And those processes would be permanent, until such time as the Galaxy is shut down.

(so between 2 weeks and a year)

(planet processes, I mean) - whereas project processes only last as long as required for the project to complete.

[quartz dust]

Processes have memory and message overhead. Plus unbounded concurrency is a concern. The advice I’ve been given by engineers I respect is it’s advisable not to go too crazy with processes

[bronze walrus]

yeah, same advice that @oak crescent gave. hehe

So each planet then needs to manage the following state:

• Projects being worked on (construction, research.etc.)
• Fleets that belong to that planet
• Notifications specific to that planet.etc.
• Bunch of other stuff

That's a lot of state to manage =\

I think I just need to dive in, hey - because a lot of that is probably just database-layer concerns.

So then the question is, how are these processes managed when database state is changed?

And even with a planet process, how do I manage when projects are meant to complete? So many questions. lol

[bronze walrus]

I was looking at potentially using this instead: https://github.com/akira/exq#do-you-need-exq

But there doesn't appear to be a way to stop or cancel queued jobs.

[oak crescent]

you should try to do it with no database at all, you just need to store the state each time you make a change to it so if it crashes you can bring it back up. you could put this in a database, use a file on the fs, dets, etc.

just because it's a lot of state to manage doesn't mean you need many processes

any actions taken should be handled by the running process, and if nobody is playing you can either change the data directly in the db, log the message to be processed later when the player connects, etc. if the player is connected have it go directly to the process so it can manage the view of the player as-needed. as far as project completion, it seems like that could just be a "completed at" timestamp that is precalculated. if there are any negative effects to it from attacks or whatever you can just change the timestamps accordingly. when the player connects you would just schedule a message for each of the builds for completion_timestamp_in_ms - now_in_ms using send_after. you can keep track of this refs from scheduling them in the state and adjust them later if need be

when the process starts it would just load up all the projects, make notifications for the ones that completed while they were offline (completion_timestamp <= now), and schedule messages with send_after for when they are actually going to complete

again if you need to do stuff async you can always spawn off a task that will run concurrently. the main concern I'd have is not overloading your process with messages, that's when you may want to consider additional processes but there's other tactics too like selective receive that are more advanced. likely you would only run into that if the game became very popular

[bronze walrus]

They wouldn't rexeive many, but I think events would be better anyway.

[bronze walrus]

This wouldn't work, too many things are happening whilst players are off-line.

[oak crescent]

why does it matter how many things there are happening?

those things just mutate state or log events out that are processed when they connect

[bronze walrus]

What I mean is - it's not just that player that is affected by changes in their planet or empire game state.

[bronze walrus]

Hmmm. Can you elaborate on this a little? What is selective receive?

[oak crescent]

you can read about it here https://learnyousomeerlang.com/more-on-multiprocessing#selective-receives

[oak crescent]

couldn't you publish out the changes though and take the same approach to mutating/logging the events across multiple players?

[bronze walrus]

I could, but this seems overly complex, in all honesty. Most of these requirements are temporary (maybe a few days' life cycle at most), and being able to keep a process running whilst listening for events just sounds very, very simple, as opposed to constantly having to update state across all these other areas all the time (which will still need to happen in many cases). Also keeping it in memory on the server is just faster.

As you can see, I'm really struggling to move away from this one concept - to me, starting a process that runs until it needs to complete, whilst listening for events that may affect it, either due to the time needing to be updated, or the action being paused or cancelled, just seems super simple. In many cases, these processes would only be listening for a few events, and otherwise not doing anything until their allotted time to conclude.

[bronze walrus]

I wanted to come back to this one last time, @pulsar oasis @oak crescent @raw maple @proud wyvern and @quartz dust , as I'm not sure I fleshed out the business requirement itself, and it got a bit whacky as we discussed. So here it is fleshed out in full:

In order for players to complete tasks, actions must be executed. Things like "start production of building x" or "send my fleet to destination y" or "start researching technology b". What makes this interesting, is those actions need to react to events in real-time that could affect them. Taking the building example, a player may have resource production reduced, which could slow or cancel the construction project entirely - so the action, needs to respond to this, and update the time at which that action will complete. Furthmore, if a player sent their fleet away, they may wish to recall that fleet at a later point - so the action associated with that fleet would need to be notified that it needs to be cancelled, and a new action would be created, something like "fleet is returning to origin in x minutes".

The actions themselves and the time at which they complete is easy, but being able to have them watch for game state changes and then manage how they should respond is where it gets tricky. I've done this with queues in the past but it becomes unwieldy over time.

As a pretty full-on example - combat engagements run for a particular point of time, and the state of those engagements is changing all the time, until the engagement has concluded (either there's no combatants for all remaining targets or a player decides to retreat or send their fleet away). As an even more in-depth requirement - other fleets may join the engagement. (I'm especially lookign forward to coding this, as I haven't seen this done anywhere for these games - yet :P)

In all cases, the actions have to be reactive to what is happening across the system.

I realised that the initial post was a bit waffly - so hopefully this helps drive ideas/discussion toward a good solution 🙂

[bronze walrus]

Hey this was a super awesome article/guide - ty! 🙂

[proud wyvern]

I still think that you need to establish a state model. I think you need to separate storing/tracking game state and rendering game state. And I think that if you want your game to be stable, then it needs to be survivable if your elixir process dies and restarts.

First, I would think about all of the things that define your game state - resource values, resource build rates, orders issued, unit parameters, etc. - and build that into a model

Second, I would recommend having services that subscribe to events, that can then do interesting things. For example, if you have a resource gather rate, and a steady state resource fee (say you have monthly tax you've negotiated to pay a neighboring civlization for peace) and your capital gets nuked... there is a process that subscribes to the "nuke" event, looks at the values that have changed, and then you have a script to see if the new value is in good standing. Basically this is a validation step - if your math that adds up stateful pluses and minuses in your credit production rate is negative, now you can do other interesting things - maybe you set a time when you hit 0 credits which you store in state and link to an event emission, which leads to logic which loads sheds cost in some predefined order. I think here, Phoenix PubSub is your friend

[bronze walrus]

You're a legend, Ty. Basically what you said is exactly how I'm thinking about it 🙂

The challenge for me is the reactivity bit - in order for them to react to game state changes, they have to be running...

Actually it does seem like you're advocating for something a little different, @proud wyvern - I'll try and steel man your point here.

Let's assume the use-case outlined at the start: I'm constructing something, and something upsets its ability to be completed.

1. I start construction of a project - this updates the game state and says that Building X is now due to complete in a few hours.
2. This reduces the total resource output by say 5 (out of 7), leaving me 2 remaining
3. The player is raided, and they lose 3 resource generation - bringing them down to -1 (technically 0).
4. Because they no longer have the resources to complete Building X - Building X needs to be paused, or slowed construction.
5. If paused, update the db, setting the new time to completion for the building.

My question is this - what process or service is handling Building X's completion? This is where Im tripping over. It's been my view that a process manages this - it runs infinitely, responds to events, and then when it's time to complete, it closes.

Said process in the use-case above, might be paused by the player to free up resources, or slowed. Both of those scenarios result in different outcomes - if it's paused, the process would be closed until such time as the construction is resumed. If it's slowed, the process stays alive, but the time to complete is extended by the required amount of time.

@proud wyvern Thoughts? Or am I just thinking about this all wrong.

To be clear, I'm not looking to use processes to manage state, but to react to state changes in real-time.

Another approach, instead of them running infintiely in a loop, is to delay the execution of a task, and have that managed? That way if the task is cancelled or its time to completion changes, I could update the elixir task?

[outer locust]

I would just make a list of counter structs and on each tick, check if the conditions are met to add time to each counter and if they area increment the counter

[proud wyvern]

I think the way this could work is that you have 1) a raid event handler and 2) a regularly refreshed passive event resolver

In this case, the raid takes places, a raid event handler is called... here you can do a few things:

1. Update the current resource value (based on previous rate and previous resource value)
2. Update the current resource rate

3. If the resource rate is below 0, set an event in the passive event resolver for the intercept of that resource at 0

When the passive event resolver is called, you have common logic that handles any case when you're out of a resource for what you load shed

[bronze walrus]

Each tick?

[outer locust]

your game is a simulation- each tick is one unit of simulated time

[bronze walrus]

I see... so you don't need a process for this, you just have some handler that listens on a topic?

[outer locust]

like your classic old school game loop is just draw frame, update simulation, handle input, loop

[bronze walrus]

This still confuses me. haha. The game is as real-time as I can manage - things need to happen instantly, as they happen

[bronze walrus]

Ah, gotcha.

[outer locust]

so if your game ran at 60 fps and you update it once per frame, each tick is 1/60th of a second

[bronze walrus]

Yeah, roger that. Well this is a server-based game, everything is happening on the server, so who knows how fast that is. haha

Even so, what you're suggesting to me says an infinite process, that exits when required.

[outer locust]

what i’m trying to say is not every problem needs to be solved by process.spawn

[bronze walrus]

No, I know - but in this particular instance, it does seem fitting. The thing is, that use-case is relatively common in the game.

What I'm wanting to avoid is querying the database constantly to check state. And to my knowledge, that can only happen in one of 3 ways:

1. Background jobs that execute at their predetermined time (which can't really be interacted with)
2. A scheduled task that can be cancelled or altered as required
3. An infinite process that exits when it's time to.

2 & 3 are the only real options, imho

[outer locust]

Sasa Juric has a good article on this topic - https://www.theerlangelist.com/article/spawn_or_not

[bronze walrus]

Yeah, I read it - and it did cover my case - their argument in my case is that yes, spawn a process.

However, considering I'd have potentially 100s of 1000s of these at any one time, many in the community ahve said "no" - so I'm trying to figure out the alternative - with help from you lovely folks 🙂

[outer locust]

you can have a single process handle the entire simulation, looping per tick and applying updates

[bronze walrus]

Yeah but the state management of that would be.... my god. lol

But maybe there's a middle-ground there... 🤔

[outer locust]

trust me, it’s easier tgan managing “one process per thing with mutable state”

[bronze walrus]

What about the scheduled tasks idea? They're not running, but they could be updated, cancelled.etc.

Sorry, I should say "delayed tasks"

Schedule has its own meaning.

[outer locust]

It sounded like from your description, the time required for things to be built isn’t strictly related to real time because there are many reasons builds could pause, to me that says managing the schedules will be a hassle

[bronze walrus]

Wouod that not be the case with a big process to manage those things though?

I guess the benefit of larger processes like that is that, I wouldn't hit process message queue issues would I?

[outer locust]

Additive counters with conditions to update is very straightforward to write - precisely you would only be dealing with your own data

[bronze walrus]

Sorry. I didn't understand what you meant by cojnters.

Also, I guess using one large process and breaking it down later if needed is easier than bringing 100s of different processes together 🤔

[outer locust]

the way i would model a thing that takes time to build is to express the time to build as N ticks. on every update pass, if the counter is not paused, add a tick to the time spent building counter. When the counter = N, your thing is complete

it’s like the active battle gauge in the old Final Fantasy games

[bronze walrus]

Let's assume I have one massive process managing 10000 players and their in progress items (potentially millions at a time). How exactly do you update it's state without getting conflicts or race conditions?!

[outer locust]

during a message handler your process can’t be interrupted

so just have a tick message handler and broadcast updates to the players when things finish building

[bronze walrus]

Ah okay, so as this big process is listening to a heap of events, it'll execute one at a time.

And then I guess re-spawn itself with the new state, once updated?

[outer locust]

return to genserver loop you mean

[bronze walrus]

Sorry, as you can see I'm extremely ignorant here - I'm diving deep into the these mechanics and architectural challenges whilst knowing very little about Elixir.

[outer locust]

no worries

all roads lead to genserver eventually

[bronze walrus]

Can you elaborate on this a little?

(within the context of what we're talking about) - maybe this is the piece I'm missing.

[outer locust]

you don’t generally just spawn raw processes in Elixir. You need an OTP compliant process of which GenServer is the simplest

the standard message handling api in genserver has you take your current state as an argument then you return the updated state

[bronze walrus]

Ah, I see.

goes and reads genserver

Ty very much @outer locust - given me some great food for thought, and it's starting to make more sense 🙂

[raw maple]

You can maybe take a look at: https://github.com/ecsx-framework/ECSx

It's for ECS, so a specific way of modeling your game engine, but still

There is the idea of ticks, and processes to handle and manage the changes per entities

• there is a cache layer and database layer

not sure how it handle clustering or crashes/restarts though!

Supposedly it should read from the database at this point