#A general question about ordering the systems in DOTS ECS

1 messages · Page 1 of 1 (latest)

devout perch
#

In DOTS ECS the systems are ordered using attributes that establish relative order between two systems. This looks quite weird. It's much easier to simply create systems and system groups and explicitly specify their order in the code and logic.

Why this approach with attributes is absolutely necessary instead of just letting us create and maintain some ordered structure?
How do you create and add / remove / reorder systems in runtime?

indigo widget
#

How do you create and add / remove / reorder systems in runtime?
you don't

#

it doesn't make sense to add/remove/reorder systems at runtime

#

systems react to data, if there is no data to them to react to they don't do anything - that's how they stop running

devout perch
#

Well, I understand what you mean, and for most situations I agree, but in general this is biased and debatable.
In most ECS-es I've worked with there was no such implication.
In my current application I would want to have that type of control -- to be able to dynamically add and reorder systems -- instead of having an overcomplicated setup with a static order.

#

But that aside, I still wonder -- what about the enforced need to specify the order of systems via attributes?
Why is this necessary instead of just letting us to normally assemble a list of systems and groups explicitly in one place in some bootstrap method?

indigo widget
#

there's nothing stopping you adding systems to a group in whatever order you want in the bootstrap

#

i would strongly recommend against it, but you are welcome to do it

devout perch
#

Why would you not recommend it though?

#

I don't see how having system order spread in a harcoded way amongst tens of files is s good idea tbh

indigo widget
#

because ideally system order shouldn't matter

#

except within a small group of systems to create a feature

#

once you hit 600 systems

#

do you think a single file ordering these in a specific way is a good design?

#

order systems within only a feature set
put htem into groups, order groups between features /if/ required (most times not required)

devout perch
#

That's why I'm saying that having it being spread among 600 files is kinda frustrating.
I'm not sure why you're saying that system order doesn't matter if having a well-defined main cycle is one of the cornerstones of a game loop

indigo widget
#

no one cared about monobehaviour orders, you'd put a few in the priority list if it was important to come first/last

#

but that's it

devout perch
#

I only ever used 1 MonoBehavior with 1 update tho.
I'm not a fan of the GameObject/MonoBehavior approach, and haven't seen it being used much in the projects that I've worked either.
It's true that VFX might update in their own Update() but not game logic

#

I kinda appreciate your input but I'm puzzled because the notion that the order in game logic and main cycle loop is not important goes against almost any project I've worked on.
I'm not sure how one can build a game with reliable behavior if the order of events in the main loop is not determined.

indigo widget
#

i'm mostly just suggesting that you don't care about system order, you care much more about group order

#

combatsystemgroup updates after aisystemgroup

#

each group might have 20 systems

#

but those systems don't care about anything else except things related to their feature

#

you might only have a dozen high level system groups

#

and in general, they'd only need to update after 1 other group

devout perch
#

Then this is certainly a new pattern, I think it should be named ECSG because now there's Gs that matter no less than any of the Es, Cs and Ss

#

Well, there was a day when Unity called their GO/MB a ECS

#

Why should I be surprised now..

devout perch
#

So I thought about it more,
And anyway, even in the workings of the game AIs or in the workings of the combat system group, I don't see how a game can work without a perfect order.
Yes there might be a few cases where there's no preference for one thing before another and the systems are only separated for the sake of encapsulation and not ordering, but that's hardly the dominant case at all.

I want AIs to first perceive the world, then evaluate and then make decisions, and only then communicate,
And I want guns to fire first, and only after that -- update the projectiles.

If it's implied that all these stages are system groups with 1-2 systems, then that's just ridiculous.
The group concept makes no sense, it's just the system concept split in two, and hey lo ECSG.

Otherwise, if the order of any of these things or their sublogic is not guaranteed, then goodbye any predictability and I'm not even talking about networked gameplay.
Take ConterStrike, StarCraft, Quake, Risk of Rain or any precision platformer or RTS or networked action game, for example, I bet their developers would just laugh at that.

indigo widget
#

I don't see how a game can work without a perfect order.
i can tell you it can because a lot games, including our own, have released without hard coding a defined order between everything

devout perch
#

Well, I think that probably puzzle games can do that, or quests, or interactive novellas, or card games, etc..
I now remember I once worked on a mobile farm game where order was also nonexistent.
Like, you can even write games in ReactJS on a website, with a Spring backend and it will work, yeah.

But it's definitely not games that have a tight action loop.
I should've pointed more explicitly that I meant that.

indigo widget
#

our game was an open world survival game with pretty much every mechanic you can think of

#

combat, buildings, turrets, ai, skills, spells, talents, survival mechanics, weather events, etc

#

600 systems from memory

#

1500 jobs

devout perch
#

Oh wow

indigo widget
#

i don't think this was just a tiny puzzle game

devout perch
#

Yeah, surely it sounds like it wasn't

indigo widget
#

i'm not saying you can just have random system orders, but a few restraints can go a long way and it's a lot more flexible once you add new solutions in the future

devout perch
#

Well, then idk what to say here. Probably you really didn't care about the order.
I don' know what was the architectural specifics of that game, probably it left some leeway for that.
And it's hard to say what were the consequences of that.
Hard to say how game design was carried out then, like, for what I know it mostly nukes the whole process.

indigo widget
#

anyway i'm probably not going to convenience you, i think that comes with experience but i will let you know that you can basically ignore unity's entire setup and do it yourself

lyric comet
#

enableable components, checking if data on component is valid or valid queries is usually how my systems get ordered naturally, no idea if thats any good to you but i very rarely use update before/after and dont have that many update groups either, doubt this is the correct way to do things but screw having 100's of update before/afters

indigo widget
#

add a
[assembly:DisableAutoCreation]
which will disable everything from auto setting up in an asmdef, then you can create and order all your systems by hand and do whatever you want

#

you can even just create a single system to control your order by calling Update() manually on the systems in a specific order or skip the update if you don't want a system to run

#

if you really really want

devout perch
#

I'm currently just reading about DOTS ECS, and this point about systems ordering is absolutely different from any ECS that I've worked with before.
So I was very curious of how's and why's.
Not that you need to convince me, I can believe anything -- no special powers needed.
Just trying to understand.

devout perch
devout perch
lyric comet
mortal marsh
harsh shoal
#

Many also surprised that structural changes are a thing (adding/removing components is best to be avoided in dots)

lean hornet
#

I have a genuine question about ordering systems - when I choose to use default group (simulate) and no other ordering attributes, will systems ordering itself regarding to theirs dependencies or they have a random order?
General question is it better to leave system unordered at all or I need to think about every one of them.
Sometimes it bothers me because I love to have single-purpose systems and even now have many of them. Afraid of the time when it will be A LOT of the systems.

harsh shoal
devout perch
# harsh shoal yeah, I'v heard a lot similiar stuff from other ECS folk (non dots ones). I'd sa...

Right, exactly, and that's the whole question: Why?
You are not answering, you are repeating my question but without a question mark at the end 😄

I'm really interested to know why is it necessary to order systems via class attributes and why the normal approach is impossible/discouraged.
So far I haven't found an answer.
I of course understand that DOTS ECS caters for some Unity-specific things, but otherwise I can't see how it's principally different from other ECSes of this particular design breed.

So is there any deep technical need for this weird approach to ordering systems?
Or is there some deep philosophical thought behind it?
Or something else?

This, as of yet, remains a total mystery. As far as I see -- for me and for anyone else in this chat.
I could, of course go into the source code and try to find the answer (if there is one), spend time on that...
But I was just sure that there was some commonly known answer, and people were not just blindly doing this weird stuff or simply leaving their systems unordered.

Many also surprised that structural changes are a thing (adding/removing components is best to be avoided in dots)
Structural changes were always a problem with archetype-based ECSes.
So since DOTS ECS is archetype-based, it seems pretty natural 😄

lyric comet
#

i think like tertle said because ideally system order shouldn't matter except within a small group of systems to create a feature is the best you're going to get until a dev explains their design ethos a bit more indepth xD JW do you have to order every system in the ecs framework your using? never looked into any that aren't dots

stoic comet
#

Yeah, it's not that you can't order all the systems in a specific way, is just that you don't need that. You just need tell the program "I need system X to run before Y" but you don't really care when they run or what runs in the middle of that. Unless you do, and in that case you just make another "I need system {} to run before {}" instruction

devout perch
#

Yeah, tertle did indeed say that.
But this is just a phrase which is not based on anything, it's not "best", it is a claim that contradicts most of what I know.
I can say as well Systems must have a defined execution order. This is vital to building an ECS design.
I can even quote someone else saying this: https://publish.reddit.com/embed?url=https://www.reddit.com/r/gamedev/comments/yctgea/comment/itr91k7?snippet=0_1168_43.
And I have worked on several projects where this was a ground truth.
And seen countless others where it was the same.
Having a 100% defined order was always mandatory for many types of games (but not all).

devout perch
#

I am genuinely amazed how much irrelevant discussion was caused by these simple questions

stoic comet
#

Ah the attributes is the "automatic" way. You can also create a group and add systems manually with the order you want.

#

I think the scripting docs cover that

devout perch
#

Yep, and tertle mentioned that already.
Originally I was reading just the docs, and as far as I have read them, they only talked about ordering the systems with attributes.
No other way is mentioned for like 50 mentions of the attributes.

#

That was what puzzled me and I decided to ask

stoic comet
#

The manual only covers the Attributes as you mentioned. The scripting docs is what you need

devout perch
stoic comet
#

Haha all the Entities docs seem like a puzzle designed to filter users that don't know how to use Entities

devout perch
#

LOL 😆
True, they are terrible.
Very hard to read.

umbral linden
#

i've never used any other ECS frameworks, but is the alternative to the attributes just a giant ordered list of your systems? Seems to me the attributes I can see/edit at a glance at the top of my system is better than that.

devout perch
# umbral linden i've never used any other ECS frameworks, but is the alternative to the attribut...

Yes, usually you have a list or an array of systems and you just populate it explicitly, like if you were adding any other kind of stuff to any other list.
You can modify it in runtime and also have some of the systems to be "group systems" that just have a similar list within.

I am not sure how is it more comfortable to have the order to be specified as a class attribute.

  • Why would you be looking at the top line of your classes at all? How often do you do that normally when you code your stuff? And why?
  • When you have 600 systems, how are you going to tell if system DucksGoQuack executes before or after the system HeySayWhat ? Your search through the attributes is going to be a O(N) operation. Now imagine you did the search, and you had to search through all 600 systems and find out that they are in separate sets of relative specifications. You might want to cheat and say "I'll group them in groups", but groups are a "giant ordered list" -- this is not comfortable according to your statement, so you can't do that.
  • How are you going to solve the case of adding or removing systems in runtime? Please note: I'm not asking how to create all systems at the start and invent a relay powerhouse of switches for turning them on and off, I literally mean creating, removing and reordering them at runtime.

You're saying all this is better than having a neat list in a single place?

stoic comet
#

You're confusing yourself with wrong assumptions. The Unity approach is really simple. Either you care about one specific system running before/after another specific system or you don't. You should only care about that if one of the systems depends on the result of the other system, if that doesn't happen then you should not care about the order. So, in the case you care about the order, you just tell the program which system should run first, and that's all.

You don't ever need to check which system runs first, because if you care about that in the first place, you should tell the program that, not ask.

umbral linden
# devout perch Yes, usually you have a list or an array of systems and you just populate it exp...
  • I don't often look at the top line, but it's a lot easier to scroll to the top of a script than open a different one and search for my system.
  • My IDE can also tell me with a click which systems reference other systems, so it's not really an O(N) operation.
  • I've never wanted to add a system or reorder it at runtime, so I can't comment on that.

Probably less than half my systems are ordered, and most of those are linked in small groups, so having to order every single system would be annoying. Also, wouldn't it make parallelization a lot harder? DOTS/Jobs does some of that automagically for you just with data dependencies, and it seems like a rigid order might break that.

I find myself actually ordering systems less as I go and try to jobify and optimize more. Trying to eliminate sync points has been leading me towards ordering things by group instead of individually.

devout perch
# stoic comet You're confusing yourself with wrong assumptions. The Unity approach is really s...

No, I care about the precise order in which the systems follow each other in the game cycle.
There is no other interpretation of what I care for or what I should care about.
Please read that again if it's still somehow causing doubt.
Because this influences a lot of stuff.
And all of this is tenfold important when you work in a team, especially a bigger one, and everyone clearly understands what works how and when.
The full clarity and determinism in such fundamental things is directly convertible into saved hours and $$$.
If this is not clear, then sorry, but I have nothing else to say about it.

And yes, specifying system order clearly and precisely in a list is exactly "telling your program".
As well as directly telling other people how the program works, in a single page.
Jumping relative links between classes to find how two systems are related is "asking".

I am absolutely not sure where does this new notion about execution order being unimportant is coming from.
And why we are having this whole strange conversation here.

indigo widget
#

And yes, specifying system order clearly and precisely in a list is exactly "telling your program".
As well as directly telling other people how the program works, in a single page.
imo this actually has huge issues for teams of developers

Imagine you have a list of systems that update in this order
SystemA
SystemB
SystemC

If i'm another developer, how do I know if System C has to update after SystemA or SystemB? Is changing it to
SystemA
SystemC
SystemB

Fine? How can I know

#

what about a system that is 20 down the order

#

my personal opinion is defining relations between system is much more stable on a large team

#

rather than defining an order without relations

#

2 years later when the developer set up the order has left your team and you need to reorganize it, how can you know the dependencies

#

I don't think anyone is saying execution order is unimportant, but the order is only relevant between systems with relationships

#

relationships are important for order and that's what the attributes provide you

#

now I think debugging tools to provide an overall view of system order and why it's order specific ways would be a great addition

devout perch
# umbral linden - I don't often look at the top line, but it's a lot easier to scroll to the top...
  • It is not (as good an argument as yours 😄, right? ). And there is no reason why you should be looking at that list often at all. It is there when you need to have an overview of your cycle or introduce other people to the project. The list is mostly used to instruct your program to work in a defined order. This order is usually determined in the program design document, so if you work on a project and don't remember what that document says, that's a problem on another level.
  • Systems should not reference other systems. This is one of the base principles of ECS.
  • Fair. You may or may not find such a need depending on the projects you work on.

As for DOTS/Jobs magic that may or may not happen: I don't know. This is exactly what I was asking about:

Why this approach with attributes is absolutely necessary instead of just letting us create and maintain some ordered structure?
I have not yet found any information about this topic.
No one in this chat ever referred to that as well.
Instead I'm getting a high dose of indoctrination about how undefined program flow is better than a defined one.
So, unless there's some precise information about that, currently we must assume that there's no relation between system ordering and any possible implicit optimizations that Unity might do.

stoic comet
#

indoctrination
Bro, people is just trying to help.

#

Maybe we haven't been good enough at explaining this stuff, but saying that it's just weird

devout perch
#

tertle, what you are saying is important, that is true.

However, a list in which systems are ordered explicitly states what system must go when. Easily readable in text.
Usually you do not reorder systems unless you understand the whole flow.
Because no matter if you have them in a list or specified relatively, if you don't understand the whole picture, any change of the order can bring consequences.
Relative specification does not provide any protection against that.

Tests should verify the relationships between systems and the passing of data.
If something fails, you see it immediately.
Also there's project documentation that specifies what happens when and why, it's what one addresses first in case of questions.

It's good if an IDE or Unity provides you aid in runtime when you can see the systems,
But this is very limited.
What if you have 10 different cycles, or different phases of the game/program?
That requires running the game and getting to some state just to see the order of systems?

Since we're discussing the "list of systems" thing now I'm not even mentioning dynamically updated lists here, it's a whole another can of worms.

indigo widget
#

I've seen a couple of users request list or priority based system ordering in the past, but in 5.5 years of dots development i believe you're the first i've seen wanting to dynamically order your systems at runtime

#

that really makes no sense to me

#

i can understand the desire for a strict order, even if i disagree with it

#

but not runtime dynamic ordering (with how entities works, maybe this makes more sense on other ecs libraries)

#

i would be very curious about a use case

devout perch
# stoic comet Maybe we haven't been good enough at explaining this stuff, but saying that it's...

Weird is the discussion in this thread. It is the first time in 20+ years that I hear stuff like this, and I don't see how it should work in real life.
(Clarification: there are games where the order is not important, as it was already mentioned, and quite a lot of them, but my question, obviously, was not about such games)

I appreciate the links to the API docs because I didn't reach them yet at the point of asking this question and was in doubt if I should have continued reading at all. Thank you guys, who have referenced them.

devout perch
# indigo widget i would be very curious about a use case

I am still researching viability of using DOTS ECS and as one of the points I need to estimate the dynamicity and flexibility of it as a framework.
Besides that being one of the points in my list that I have to somehow fill in, it is true that we have used that with other ECS systems, but it might be not so easy with DOTS ECS.

The use case for such scenarios is having dynamic worlds which can be updated later or enriched by mods/plugins.
Modding should be able to create and insert system into any point of the main cycle of the application.
It is possible to provide only fixed insertion points, but it is limiting and at some point mounts too much complexity.

Also, there is a notion that it is not so simple with Burst-optimized code.
But we believe it's possible to build an API that works.
Also we are looking into Burst scripting
For example, one of the projects realizing Burst scripting:
-- https://github.com/nijnstein/Blast
-- https://forum.unity.com/threads/runtime-ai-scripting-in-burst.1158443/

GitHub
GitHub - nijnstein/Blast: Blast Script Engine

Blast Script Engine. Contribute to nijnstein/Blast development by creating an account on GitHub.

Unity Forum
runtime ai scripting in burst

hi, anyone any experience with a light weight scripting engine that can be used from burst code with only minimal symantics? Im currently developing...

harsh shoal
#

to be precise order is resolved at runtime via reflection

#

It just iterates all system types (cached) and finds out which fit world, to which system group they belong and then they get sorted based of local to group attributes

#

Also, there is systems window which shows you the order of your system in a list manner

#

Which is very convinient when designing data flow or debugging

#

On top of that that window even supports finding systems by type used

#

you type SomeBuggedType in search and it filters list to only systems that use it in an actual update order which is sick imo

devout perch
# harsh shoal Mods just reference your assemblies and use attributes to inject, that's about i...

Thank you for these great insights.
I always welcome additional information and thoughts, but please note:

I am not looking into outsourcing the system instantiation and/or ordering logic to Unity or DOTS ECS magic.
No clear technical reason for doing that was ever referenced.

It does not matter what compensatory mechanisms Unity provides to work around the obscurity that arises from the attribute-based approach and/or lack of clearly controlled ordering.
The tools like a visual system list with all kinds of filters and search are pretty mundane for ECS, please stop portraying them as some kind of a miracle.
Besides, they do not have any relation to attribute-based ordering, and as such to the OP question, at all.

We are not looking into modding exclusively with C#, as you might have guessed from the previous comment,
So relying on C# reflection is out of question.

harsh shoal
#

I guess if you are not satisfied with dots solution for world bootstrap - there are tools to create your own. You can write all of this code from scratch to fit your needs.

But I really think you just need to see how it works first and feel it, before doing something as radical as this.

devout perch
#

What we are doing is not radical, it's pretty standard.
As it was answered, DOTS ECS does support the usual programmatic approach, it just has weird defaults.
So there should be no problem.

harsh shoal
#

some of them are even internal

#

but internal is not a problem, because you can still access those

#

but it's going to be a pretty big code file in any case

devout perch
#

Custom bootstrap and full setup is possible, I've already checked the relevant docs.
No problem, there are going to be many many big code files in the project.

#

For all it's existence and bumping up
This discussion has not yet attracted any example of hidden technical implications related to the OP questions in Unity DOTS ECS.
It was also revealed that the traditional programmatic approach is fully supported, although a bit complicated.
I think we have actually reached all the conclusions we could have made.

final crescent
#

Very interesting talk. I’d be interested in the use case, where the order of systems really is necessary to determine, because that would mean, you have to react or act in exactly the same frame on a large scale of different operations. While data orientation awaits data to be there, which might have been generated a few frames ago and is still fine. Hence probably the reaction, that strict order does not matter usually. As said, very interested in details, not to fall in a similar trap in my ECS future me.

harsh shoal
#

in DOTS unity decided something different

#

and I personally think it works better, than asset defined order

#

especially it shines when you add new libraries with systems and you don't even need to change anything as you import

#

because library dev specified all update constraint on his side

devout perch
# final crescent Very interesting talk. I’d be interested in the use case, where the order of sys...

I don't have much to add besides what was already said in this thread.
Also I'm not going to address the misconceptions introduced in the latest comment by Issue.
The OP was answered and this thread has gone way off topic.
If you have any other questions it's best to ask them in a separate thread.
If you are interested in general ECS discussion, not DOTS ECS details, then it's best to ask elsewhere completely.

versed quiver
#

This post is very interesting, I found unity comes with automatic random sorting when writing system order manually, and found that the default EnableSystemOrder is true in ComponentSystemGroup, if adjusted to false than systems can be manually sorted. Of course, most of the time order is something that people shouldn't care about, like tertle said, if a project has a lot of engineers, the order that needs to be determined is to make things obvious, if a project is small, the system as a pure function of things, should be able to clearly express what a system in the module does, manually determining the order can quickly rearrange the order, I think it should depend on the conditions 🙂

pallid thistle
#

one reason i am a proponent of the current constraint-based ordering scheme is that it leaves the door open for systems updating in parallel within the same world someday(tm). if the default is an explicit linear straight-ahead order, you lose the information about which systems have ordering constraints if you want to go parallelize