archived-dots

[sturdy rune]

Yip

[velvet panther]

am currently using static persistent nativeArrays as "blobs" but feel (not tested, don't know) that blobs might be faster for this kind of reference file usage between jobs.

[left oak]

That might be the case. I'm no expert on Blobs, but they might be superior for that purpose, depending on too many factors for me to just generalize

[velvet panther]

It's only my speculation, based on the fact that they're constantly discussed as being ideal for LUTs.

[left oak]

I would say "go for it", but your caveat that you want to do this outside of normal Entities usage gives me pause

[velvet panther]

Cheers. My main issue with ECS is the extra time to compilation, ruining my iteration flow. I'm super enjoying stackalloc usage without Burst and Jobs, whilst getting everything working, because adding Jobs and Burst absolutely ruins the iteration workflow.

[haughty rampart]

if you have a 'problem' with your iteration workflow, just turn of burst compilation. it will then run your code just fine as normal interpreted c# code. when you build your game you can enable burst compilation again. don't make up problems when there aren't any.

[velvet panther]

that's exactly what i was saying I do!

[sharp dragon]

You won't use ecs cause compilation time, but you use burst... 😕

[spice steeple]

any way to unload a subscene from a file that is not part of ComponentSystem

[velvet panther]

you're not paying attention... I don't use Burst, ECS or Jobs when coding for iterative purposes, and have almost no other Packages installed, either. I code in a style (often) that's later suited to turning on Burst and Jobifying for performance gains once the work is proven and done. Was this hard to follow?

[velvet panther]

the next time you want to be like this, do it somewhere else.

[digital kestrel]

a lot of heat last night 😳

[gilded glacier]

any good ecs practices ?

like whether a system should be able to disable other system (or interact with it in any other way) ?

[sinful cipher]

Im not exactly an expert, but ive done some complex stuff in DOTS.

Hard to say for me without knowing specifics. What are you trying to achieve?

[gilded glacier]

I have a command that spawns objects and stuff, which means those objects need to be loaded, initialized, placed, etc.

basically a sequence of complex stuff, but the thing is I have one system that queues for the command, and I intend to have this system controlling other "subsystems" that each does their designated part in the sequence

and I don't want those subsystems to run all the time, because they work with relatively large entities

but I ask generally, because there should be some practice for this kind of interaction between systems

(not found any book on this topic)

unless it's completely fine to go rogue in ecs ?

[sinful cipher]

So your thought here is:
Let system A disable system B if system A determines there are no (eligable) entities for system B to operate on?

[gilded glacier]

a bit simplified, but yes

[safe lintel]

i feel as if your system should generally only run if there are entities to process, would that system end up repeating its work if left not explicitly disabled(ie double work)?

[sinful cipher]

I myself use 'flag components' for this purpose. I recall seeing a presentation featured by Unity about a studio doing something similar. I can try to look for it.

So regarding the flag components:
Say I have a piece of terrain of which the shape is modified.
I attach a TopographyChangedComponent to it.
Then some MeshUpdateSystem only iterates through terrain piece entities that have this _ TopographyChangedComponent_ , and removes it after its done.

[safe lintel]

like the whole architecture is generally built around entity queries and systems just run or dont run, ie like the transform system. if your platform is mobile then maybe want to handle more explicit disabling due to possible performance issues but that was with 300 or so systems which may or may not be way more than what you have going on

[sinful cipher]

This is also somewhat simplified, but should convey the basic idea.

[gilded glacier]

yes

[gilded glacier]

yeah, about that ...

I've encountered the issue that my entities are quite large

and also found some benchmarks that using a component with bool or something in it is more effective for that case

[safe lintel]

anyway I sorta do the same but reverse order of what onelemonarmy does, so in certain instantiation cases on my authoring conversion, I add a tag like "PlayerUninitializedTag", so a specific system catches that and then sets up the player, removes that tag and is never processed again unless another player is spawned from scratch

[gilded glacier]

but then, I cannot queue for just certain entities; I have to queue all of them

[sinful cipher]

...large? As in, literal bytes per entity?

[gilded glacier]

with the amount of items in buffer, plus additional components, the final size is up to a few kBs

[safe lintel]

with more thought, how do you handle not processing those entities again? ie

• instantiate x amount of entities
• system(that you are disabling) processes them. you disable the system
  what happens when you spawn another batch of entities of y to be processed? if you enable that system again how are you currently handling not reprocessing x?

[sinful cipher]

How exactly did you determine that your entities are large? Are you running into actual performance problems?

Also, what kind of data are we talking about. Like, are you keeping all the vertices of a mesh in a buffer for each entity or something? (just an example)

[gilded glacier]

I only did some calculations - took the estimated count of entities that can be processed at a given time, how much memory is needed to be allocated for all the components and buffers, and got a number that concerns me a bit

that's why I'm looking for any solution

but it confuses me more that I don't know technical details of how unity ecs is working with memory, caching, and stuff

[gilded glacier]

voxels in a 3d grid

[sinful cipher]

Man I'm just guessing my way through all of my problems most of the time lol. Usually I end up applying some trick or structure based on what I've seen elsewhere, or what I reasoned should work without a doubt based on what I do know.

[sinful cipher]

Is your 3d grid broken up in "chunks"? Say 32x32x32 chunks or something.

[gilded glacier]

that's what I want to figure out
my current idea is to have a system that enables/disabled those subsystems (all the chunk entities will be queued, but only once in a while, instead of with every update)

[sinful cipher]

Oh I think Im starting to understand what you're getting at. I have a similar issue with my terrain pieces where I have to generate normal maps for each patch.

[gilded glacier]

each entity representing a single voxel chunk; the amount of voxel chunk is indefinite

[sinful cipher]

This is what I did:

• I determine the terrain patch currently occupied by the terrain "observer" (e.g. the player or camera).
• This patch is queued up for processing, as well as all patches that are directly adjecent to it.
• Next update, all processed patches now queue up their neighbouring patches, but check against a Dictionary/HashSet (whatever is applicable for you) so that no patch is processed twice. Note that this requires a system in which every patch (or in your case, voxel chunk) can be referenced with some kind of procedural ID (x,y,z for your case should work).

So this ends up looking like some kind of 'wave' that propagates from the starting point (where the camera/player is) up to some maximum determined LOD range.

@gilded glacier

Again, this is simplified, and there is more to it. But this is the basic principle.

And you can have a minimum LOD distance at which chunks are processed at all costs, so that for example objects will not fall through terrain if the terrain collision mesh was not generated yet.

Or am I talking about something totally different from what you're concerned with?

[gilded glacier]

so it looks like my approach - having entities representing chunks/patches, but data of these entities change over time sequentially

[sinful cipher]

Yes. Sequentially, and in order of priority (in this case: closer is more important, with a minimum range in which chunks are updated no matter how long it might take).

[gilded glacier]

but if you have a system for each step, and don't make any structural changes that would allow those systems to queue only for those entities they're supposed to process at the time

you somehow have to ensure those systems do their work in predefined sequence (like system A, then system B) at specific time

the easiest solution is to loop through all entities every update and check, whether they require some work by a system

[sinful cipher]

This is definitely the easiest, but will start getting relatively expensive if you have 1000s of entities.

By the way, here is some insight into how memory, entity and components work (although maybe you already know this):
Entity size (as the sum of sizes of its components) do not affect how performant a system is in iterating through them. Only the size of a component itself determines this.

So say you have an entity with 20 components, each very tiny, and your systems only need about 2 or 3 of these components at a time, this should be fast.

However if you have an entity with say a few components that are all very large, this changes. This will be slower.

[gilded glacier]

I suppose a buffer is considered a single component

[sinful cipher]

Entity-component systems in memory are, in a sense, laid out where each component type resides in its own 'array'.
Array of Transforms.
Array of Velocities.
Etc.

[amber flicker]

*with the caveat that smaller components may lead to more fragmented chunks, which may lead to worse performance

[sinful cipher]

Oof, this I cant really comment on. I dont have much experience with dynamic buffers. Can these be of arbirary size, or do you always need to specify a maximum capacity?

[sinful cipher]

Yes, this is a fair point.

@gilded glacier I honestly dont know how dynamic buffers look like in memory

[amber flicker]

Essentially yes. Either it has a fixed size or else the data itself is held in managed memory.

[sinful cipher]

Can you have a component containing a dynamic buffer that can be of different size for different entities? Or can it only be the same for all?

[amber flicker]

It can only be the same for all - unless its capacity = 0 where upon the data will reside in managed memory

[gilded glacier]

the maximum size always has to be defined, but if you exceed it, the buffer is then moved into a managed memory or somewhere outside the chunk

I can make a loop that checks flag of mark components of an entity, save indices of those entities, and use it to access entities that require processing later in the system

[sinful cipher]

I believe I do something similar. Its been a while since I worked on it, so I cant say for sure.

[amber flicker]

I didn't really follow the earlier conversation (so this may not be that relevant). In general I agree with thelebaron's sentiment that a system should run whenever it's required data exists. That said, at least at the moment, it can be a pretty good optimisation when dealing with low entity counts and/or many jobs that sometimes don't run etc to enable & disable ComponentSystemGroups. It helps avoid dependency chain stuff etc.

[sinful cipher]

I never enabled/disabled Systems for this purpose, so I cannot judge for @gilded glacier whether this may give rise to complications later down the line.

[sinful cipher]

What I can say however is that I have not encountered any official resources from Unity where such a trick is utilized. I always encounter the 'flag component' design pattern instead.

[amber flicker]

(ref here if interested https://forum.unity.com/threads/very-poor-performance-on-mobille-when-having-a-lot-of-system.1184056/#post-7591222)

My point was only that sometimes doing such manual system activation can be useful.

[gilded glacier]

enabling/disabling systems is possible (although I'm not sure whether another system should do that, because that would make a dependency), also I read that singleton entity existence can be used for a system as a requirement for run

[sinful cipher]

The singleton thing is something I hadn't heard of or thought of before.

[gilded glacier]

this is a rough realm, we really need a book or something

anyway, thanks for all the insight, the complicator in my brain should be satisfied for a while

[sinful cipher]

This is really hard work yeah. All in the name of sweet sweet performance.

[gilded glacier]

that's our punishment for using experimental technology

[sinful cipher]

Well, in all fairness, the whole ECS concept isnt new at all. The guys at Guerrilla games (studio behind Killzone and Horizon Zero Dawn) have been using a component-based architecture since way back for their first game (Shellshock: Nam '67, not Killzone lol) in 2004.
Here's a more recent presentation by them regarding the topic (in context of Killzone 4): https://d3ihk4j6ie4n1g.cloudfront.net/downloads/assets/Rouwe_Jorrit_Threading_Entity_Update.pdf?mtime=20200402093015&focal=none

Although of course, yes, the API and underlying mechanisms that Unity provides to us to implement ECS is very new and experimental.

Other examples of ECS implemented in AAA games:

• Overwatch: https://www.gdcvault.com/play/1024001/-Overwatch-Gameplay-Architecture-and
• Deus Ex? Although I cant remember which one.

Although sadly I never found a lot of resources regarding more complex things such as terrain generation & synthesis

[remote crater]

Week 5, still can't destroy an entity. Any help? I reduced it to two lines, one in each systembase: https://forum.unity.com/threads/entitycommandbuffer-isnt-happy-when-destroying-an-entity.1172360/#post-7549981

[north bay]

Two things that you can try out if you haven't done so already:

1. Instead of using ecbs = World.GetOrCreateSystem<EndInitializationEntityCommandBufferSystem>(); try: ecbs = World.GetOrCreateSystem<EndFixedStepSimulationEntityCommandBufferSystem>();
2. Make sure DroneFlyClass updates before DestroyNowClass by adding the UpdateBefore attribute to your DroneFlyClass [UpdateBefore(typeof(DestroyNowClass))]

[karmic basin]

https://docs.unity3d.com/Packages/com.unity.entities@0.17/api/Unity.Entities.SystemState.RequireSingletonForUpdate.html

[remote crater]

Thank you kind sir. The helpful are the Kings of the next life. God bless you. I'll try those!

[remote crater]

At first glance, it does appear that you solved my problem.

[robust scaffold]

Just upgraded my project to 2022.1 Alpha and.... it works. Surprisingly well. Better than 2019.3

The UI doesnt need constant reimporting to work. Really nice.

[unborn totem]

you can work with Entities/Hybrid Renderer/etc in 2022.1 Alpha?

[safe lintel]

generally seems like the incompatibilities have been fixed but some lingering issues. animation isnt compatible, also they will most likely ignore and close any bug report until entities officially gets another release

[unborn totem]

has kMaxGPUPersistentInstanceDataSize ever been exposed as something configurable somewhere since Hybrid Renderer V2 came out?

i always have to go into the package code to edit it to be larger in order to get all my entities to render

[safe lintel]

might be worth bringing it up again, they coulve forgotten but think it was their intention to make it editable

[unborn totem]

they do say this on the forums

Yeah in 0.10.0 and up we have a system to automatically increase the size of the buffer. You should not need to manually tweak the numbers any longer. The only hard limit we have is the limit is the 1 GiB set internally in Unity for ComputeBuffer size.

In your case it looks like the allocation fails due to fragmentation and/or already very high usage? Is it possible you are leaking renderable entities somehow? 1GiB of entitiy data on the GPU is quite a lot.

[safe lintel]

if you can repro it, im sure they'd appreciate it

[unborn totem]

i was on a 0.5.0, project I haven't opened for awhile, so trying to find my sea legs again about what versions of things are important now

[safe lintel]

ah, guess you havent updated then?

[unborn totem]

going to look into doing that now and hope it doesn't break absolutely everything

[coarse turtle]

the one issue I've had with updating & testing to 2021.2b/2022.1a w/ entities so far is, sometimes clicking an entity in the EntityDebugger or DOTS editor might cause the editor to silently crash, nothing in the crash log for me to understand why it crashed tho 😦

[safe lintel]

ouch, that sounds annoying. ill personally be waiting to upgrade, could strip out animated stuff as its just tests but ive waited this long, hopefully its before the end of the year 🥲

new release would make for a nice thanksgiving 🦃

[robust scaffold]

Yea. I just use Entities and the Debugger. There's a lot of errors behind the scenes in the Collections package in particular. I havent updated it because then it would break Entities itself

[coarse turtle]

Same

[deft haven]

How to create "Definition" for ECS? like Scriptable Objects for Object Oriented.

[sinful cipher]

🤔

[tribal pollen]

Any ECS news for 2022?

[devout prairie]

🤷 kinda..
#archived-dots message

[north bay]

#archived-dots message

[sinful cipher]

Is it just me? Whenever I read Joachim Ante's name or see a picture of him, all I can think of is him being on stage saying the beautiful words over and over again:
"Great performance."

I see a picture of him and i go: hey its Mr. Great Performance guy!

[digital kestrel]

I see his pink shorts on stage and think ... ultimate power move

[safe lintel]

I miss the unites, they were a good incentive for unity to actually show whats upcoming.

[spice steeple]

For some reason, my Player object only collides with my subscenes when the subscenes were previously opened in the editor before starting the game. otherwise, the player just falls through them like there is no collider attached to the plane.
does anyone know what is going wrong there?

if i can't figure this out I am switching to Godot engine

[safe lintel]

can you reimport them? update any converterversion on your scripts if you changed the code?

or try dots>clear entities cache

[spice steeple]

will do

[golden ridge]

What is the current "official" way to run job in parallel with Entities.Foreach. I've seen people use IJobPararallel but that's deprecated.

[left oak]

ScheduleParallel() -> https://docs.unity3d.com/Packages/com.unity.entities@0.17/api/Unity.Entities.SystemBase.Entities.html#Unity_Entities_SystemBase_Entities

[golden ridge]

What's the difference with .Shedule?

[left oak]

Read the link: it parallelizes by chunk

[sinful cipher]

A question regarding principles/practices: Striving for zero memory allocation per frame with DOTS is not realistic for a large and complex game, is it?
I used to be able to do this, but with current up-to-date DOTS theres always some memory allocation going on here and there. Not a lot though.

[golden ridge]

I guess it schedules them at the same time compared to independently. But would that not be slower that just having one job per entity instead of entity query?

[safe lintel]

@gusty comet there is but its super work in progress, imo wait for the next release before diving in

its in the pinned messages though if you want to investigate it

[mystic quartz]

Blugh
Instantiating an authored GameObject as a prefab? Would be kind of pointless to have an editor and not be able to use it for creating new Entities.

[digital kestrel]

no but there is a way to bake skinned animations into textures and instance render them

[sinful cipher]

I worked alot with DOTS but I never really looked a lot at what other people are doing and what common practices are.

I'm guessing what I'm doing here is not uncommon?

And is there a good publicly maintained repo where similar things can be found? I feel like what I'm doing here must have been done a thousand times before.

So what I'm getting at here is: useful extensions of existing DOTS-related classes.

It doesnt feel comfortable to be doing so much guesswork for so long, even though it does work in the end.

I also find myself often holding on to UnityEngine.Object instance ID's just so that I can retain these in Components and use them to refer to the corresponding UnityEngine.Object at some later point.

[sinful cipher]

This feels very janky. But so far I haven't come up with something else that is still fully Burst-compatible.

I end up with weird composites of Native Collections. It works, but doesnt feel 100% right.
Am I the only one?

This kind of crap works and is super fast, but omfg its almost too complicated.

So many collections just to sort and move data around

It starts feeling weird and off.

I have noticed a lot of the complexity arises from trying to associate GameObjects with entity data. Like for example when you're done generating a mesh for a piece of terrain on the DOTS side of things, but then need a GameObject with Renderer and Collider to represent it in the scene.

(Although maybe I'm trying too hard to squeeze out all the performance I can..)

[coarse turtle]

sometimes I feel the same way with writing my imgui framework. I just keep looking at the burst inspector to try and optimize the asm generated 😅

[unborn totem]

Has anyone got this error before when trying to render a lot of Entities with HybridRendererV2?

Maximum Hybrid Renderer batch count (65536) exceeded.

it looks like in code it's meant to be resizing an array everytime it happens

{
    if (!(m_InternalIdFreelist.Length > 0)) Debug.Assert(false, $"Maximum Hybrid Renderer batch count ({kMaxBatchCount}) exceeded.");
    int id = m_InternalIdFreelist[m_InternalIdFreelist.Length - 1];
    m_InternalIdFreelist.Resize(m_InternalIdFreelist.Length - 1, NativeArrayOptions.UninitializedMemory);
    Debug.Assert(!m_SortedInternalIds.Contains(id), "Freshly allocated batch id found in list of used ids");
    m_SortedInternalIds.Add(id);
    return id;
}```

maybe it's happening for me, just doing so very slowly, not sure exactly.

I have over 2,000,000 entities, so... heh

makes me wonder if this dynamically resizing array is quite slow when I know it's going to be resized a bunch of different times before it reaches the size I want it to be

[unborn totem]

i believe DOTS is skipping 2021 entirely and going to 2022

there has not been a lot of external communication

i think this is the latest post known about DOTS https://forum.unity.com/threads/is-dots-being-abandoned.1183621/#post-7594702

I just made a build with 6 subscenes that each have 10,000 GameObjects that get converted into Entities

does anyone know why my build's .entities files would be so huge?

as far as I am aware, each one of my 60,000 entities (10,000 per .entities file) contains the following ComponentData

it's it possible each one of my entities is 16.181 kB in size inside of these files?

feels like a lot of those components aren't even something I want, but they're getting added for the sake of the Hybrid renderer it seems

[velvet panther]

These issues are extreme in DOTS/ECS, Burst and Jobs. And present to a lesser extent in SRP materials. And Scene management. And Networking. And Audio. And VFX performance. And post processing. It's part and parcel of Unity's user base being shaped by Unity's claims, and the lack of rigour in those that make Unity for its users -- because they don't use Unity.

JacksonDunstan.com was almost single handedly doing something about this.

[hollow sorrel]

maybe it's also storing the full mesh in there?
also that file size suspiciously lines up with 16kb per entity (1 entity per chunk)

what does your chunk layout look like in the entity debugger?

[half jay]

any tips what can be wrong with this system? I have this error

ArgumentException: An Entity index is larger than the capacity of the EntityManager. This means the entity was created by a different world or the entity.Index got corrupted or incorrectly assigned and it may not be used on this EntityManager.

in this system

[coarse turtle]

try scheduling with dependencies passed into Schedule? Maybe a different job is causing some structural changes while that current scheduled job is inflight

[unborn totem]

I'll be honest, I don't fully understand the GUI for chunks or if this is good or bad

[hollow sorrel]

ahhhh

ok yea that explains it

each of those 60000 entities gets their own chunk mansion to themselves

[unborn totem]

what determines that?

[hollow sorrel]

and since subscene serialization is just packing chunks 1:1 into binary it explains the filesize

it's probably because RenderMesh is a sharedcomponent

so each one of those entities have a different rendermesh

and sharedcomponents with different values get placed in different chunks

ideally you'd have your entities share the same RenderMesh if they have the same mesh

are you doing procgen?

[unborn totem]

hm, i created scenes full of GameObjects that get converted into Entities as Subscenes

as far as I know, every single one of them use the same mesh

[hollow sorrel]

hmm

that is weird

same material instance too?

maybe it's a different sharedcomponent causing it but from that screenshot not sure which one it'd be

[unborn totem]

same material, yes, but I am using Hybrid Instances to pass different values to the material that are essentially unique to each Entity (there are definitely times where the values are the same though)

example of what I'm talking about

how do you tell if a component is a shared component or not?

oh is it this little icon?

[hollow sorrel]

not sure what that icon is, maybe?
materialproperties should be fine though

[unborn totem]

oh, well it looks like it tries to communicate "shared"

that's what that triangle icon means on mobile devices

yeah I think it is what that icon means in the Inspector

i noticed there are some other shared components the Hybrid renderer adds

this one in particular seems like it generates unique ulong values

that was me just moving around to different entities in the debugger

not sure of the inner workings as to why

looks like it may have something to do with transparent shaders

    Entity entity,
    EntityManager entityManager,
    in RenderMesh renderMesh)
{
    if (IsMaterialTransparent(renderMesh.material))
    {
        int entityId = entity.Index;
        var hash = new xxHash3.StreamingState(true);

        hash.Update(entityId);
        if (entityManager.HasComponent<SceneSection>(entity))
        {
            var sceneSection = entityManager.GetSharedComponentData<SceneSection>(entity);
            hash.Update(sceneSection.SceneGUID);
            hash.Update(sceneSection.Section);
        }

        uint2 transparentPartitionValue = hash.DigestHash64();
        entityManager.AddSharedComponentData(entity, new HybridBatchPartition
        {
            PartitionValue = (ulong) transparentPartitionValue.x |
                             ((ulong) transparentPartitionValue.y << 32),
        });
    }
}```

looks like there may be a pre-compiler flag I can set to stop this called DISABLE_HYBRID_TRANSPARENCY_BATCH_PARTITIONING

though adding it to my Project Settings seems to do nothing

there we go, i think i got it

oh this is only for 10,000 not the 60,000 btw

there we go, back to 60,000

so now my .entites files should be 1500 * 16kb right?

[hollow sorrel]

@unborn totem yeah that looks correct

[unborn totem]

cool. I wonder why it maxes out at 40 entities per chunk

size of the entities I suppose?

i feel like a lot of those components on my entities put on there by the hybrid renderer aren't really necessary in my case, but who knows

[indigo atlas]

Hello, I am having a, probably simple, issue but I am struggling to troubleshoot in online. I am trying to set a position for an object to move to if it is idle, but setting the position AND status means the position is 0,0,0. If I set the position continuously it works. If I set the agentStatus after a short delay then the script works as intended. My question is why, in this code below, could setting the agentStatus mean the destination is not set? Thanks for any help

{
  actorNavAgent.destination = claimablePositions[entityInQueryIndex];

  // including this line & the .destination is always 0,0,0. This is only place .agentStatus is set
  actorNavAgent.agentStatus = NavAgentStatus.Moving;
}```

[unborn totem]

sorry I don't know anything about the nav agent system

is that part of DOTS?

[remote crater]

Standalone renders about 15x as efficient, watch 6000 entities with Ai script + collider + HDRP render at almost 60 FPS: https://www.youtube.com/watch?v=9UmanLjAsdQ

Thank you everyone for helping me. I'm almost ready to release MOBA. Please pray my living conditions goes well. I have 20-60 hours to release, but I may have to move for the 4th time this year after my grandmother passing away and my family turning on me hatefully due to inheritance greed and covid fear.

[indigo atlas]

It's a component I made with two properties. claimablePositions is a native array, and the code I posted is inside a Job. My issue is the destination for the actorNavAgent never seems to get set if I set the agentStatus, despite I am setting it in this same block. I believe the issue is either the claimablePositions array value has not been properly set when this code runs. I'm going to fix this morning and will post my findings.

I can post the OnUpdate method somewhere if anyone cares to have a look

[indigo atlas]

The issue is I am assigning claimablePositions based on positions of things I am instantiating, but since the instatiating happens after the jobs, on the main thread, they do not yet have their positions.

[velvet panther]

All the very best Jim! Good luck, and Godspeed!

[remote crater]

Thanks bro, I need all the well wishing and prayer I can get. Super God bless ya!

[velvet panther]

Will pray for you! Make sure to focus on the real world as much (or more) than the digital one, at a time of stress and distress and grief. Give them all your love, all your patience, all your energy. Code waits.

[tribal pollen]

This might sound a bit odd... but is there a way to check component data of an entity from inside a monobehavior for that entity?

[karmic depot]

I assume you're not using a subscene for conversion?

[tribal pollen]

nope

basically nothing is in my scene by default, just a single setup script.

[karmic depot]

if nothing is in your scene, what do you mean "monobehavior for that entity"

[tribal pollen]

well I'm considering making a prefab of an object and have a mono on it

[remote crater]

yes

I do it all th etime

sec, let me get you code Ika

DOTS_MOBA_BOARD dmb = GameBoardModel.entityManager.GetComponentData<DOTS_MOBA_BOARD>(e2);

So DOTS_MOBA_BOARD is my entity data component

entityManager, you get that somehow

and Get ComponentData <type> (Entity from);

Boom then you just dmb (or whatever your var is called) dmb.variable in the DOTS_MOBA_BOARD struct or whatever you EntityDataComponent is

I actually mirror my entity off my player controller

Its the proper design pattern way

Then if I change the player controller, I only change once for my GameObject mode and Entity mode

I'm almost ready to do my big tutorial on youtube to make DOTS ez pz

Let me get my MOBA out this week or next first tho

[cerulean pulsar]

There's no formal relationship between monobehaviours and entities except during conversion. Any runtime relationship is custom. Changing your question to, "Is there a way to check component data of an entity from inside a monobehavior?" then the answer is yes, like Jim said, use YourWorld.EntityManager.GetComponentData<YourComponent>(yourEntity) . You can find your world in Unity.Entities.World.All

You can get a reference to your entity from a monobehaviour by tagging it and using YourWorld.Systems[0].GetSingletonEntity<YourUniqueTagComponent>() or by using an entity query

[remote crater]

Wow, I get even more performance if I don't launch a standalone debug build: https://youtu.be/dXyLOsHiwxM Now over 9,000 entities with scripted AI, collisions, rendered at 60 FPS!

[heady vortex]

I'm having an awfully hard time trying to understand DOTS in general

Does it basically revolve around multithreading your code?

[unborn totem]

Part of DOTS does that, yes.

Entities is another part. That part is meant to replace GameObjects one day. Though it is more complicated than that.

Biggest hurdle is that the editor has almost no GUI set up for any of this. There are some custom windows, but nothing as powerful as the GUI for GameObjects and Monobehaviours

[heady vortex]

How far into math do I need to know to work with DOTS?

[indigo atlas]

I don't think you need much maths to get started, it's more about how you think about your data and read/writing it. Then for me I spent time with the examples and watched a talk by Mike Acton at Unite which helped.

[velvet panther]

If you aren't absolutely sure you need it, to the extent you're thinking of using another engine for performance gains, then you don't need to think about DOTS/ECS, Jobs or even Burst. And even then, it's best to go up the path, from Burst, to Jobs, to ECS... as you may not need more than Burst, and if you do, you might only need Jobs. ECS is in molasses, unfinished, undocumented and undergoing massive changes, avoid at all costs.

[heady vortex]

Is burst compatible with my regular old unity code?

[velvet panther]

Yes, to a large extent, you can just add Bursting versions of your algorithmic code, wherever you might need a ... wait for it... burst of performance.

[heady vortex]

why isn't it the default compiler then, if it's so performant?

[velvet panther]

It limits what you can make. It doesn't like objects and references, and preferring value types (structs) and standalone, strictly inline functions. It's like using only the invisible C part of C#

[vagrant surge]

burst is fast because its limited vs normal c#

those limitations basically turn it into C

C is fast

[heady vortex]

ah I see

[velvet panther]

The best part about Burst is that you can use it in only the bits of your code that need to be faster. In most games you'll not need it, as graphics rendering (especially lighting and post processing) will likely be 10x slower than your game logic.

However, if you're wanting to custom make some special effect (visual or geometric moving, or something with lots of things being done in a single frame) then Burst can be great for that.

But, before then, I'd suggest taking a good long look at the Particle System (old one, known as Shuriken) because it's highly capable, and can be coded to do most any kind of visual effect that it doesn't do out of the box, and is aware of physics. It's a lot more capable than folks generally think it is.

"Jobs" is good for things that take a few frames to do, as it can spread a load across all the cores that aren't being used. So if you have a loading disaster that's taking too long to convert or decompress models and apply complex structures, then you can get as many cores as a machine has to speed that up, sometimes quite significantly. This is not simple code, and kind of requires that you understand Burst to get the most out of this, and has very few real applications.

ECS sits on top of Burst, primarily, but also Jobs, in providing a means to shape all your game as systems that Jobs and Burst can iterate over (hopefully efficiently) so that you can have 10x or 100x the objects animated and responsive in your game. However it requires you completely rethink how you make not only your objects, but how you deal with them, in terms of Systems (this is the S in ECS).

[heady vortex]

hmmm alright that sounds pretty cool

can you give a few basic examples of multithreading with jobs

like use cases

[velvet panther]

I'll give a very specific one, and hope this helps, and then ask me again if this isn't useful: I make sounds, that use quite complex shapes, to make the nuance of the sound, and it takes quite a deal of time to draw out that waveform, and I want to make many of them, somewhat dynamically, when you upgrade your engine in my game. So I "draw" those sound waves with Jobs, as that can spread the drawing load (many algorithms of sine/cos maths) across several cores, and get it done much faster than on one core, and off the main thread, so the game appears to keep running smoothly while I render this sound file.

This might take anywhere between 4 frames and 20 frames to do this rendering of the audio files, in the background, and then I know when it's ready, and can begin pulling from these "files" of audio information in realtime (they're really just arrays of complex audio).

[heady vortex]

So for example if I were doing a networked game. When the data comes in from the server it has to turn bytes back into values etc. Would it make sense to split up this operation using jobs? Like split it up between two threads?

[velvet panther]

This is what the guy from DOTSNET is working on, a set of compression and decompression algorithms that use Jobs to do it better and faster, so yes, absolutely!

He's initially doing this in Mirror, which is MonoBehaviour based, and he's jobifying the compression, so you can follow along as he's doing it. And Mirror is astonishingly well documented, in and of itself, before you get into looking at what he's doing with the Jobs stuff for compression.

[heady vortex]

If I'm not mistaken, you can't update positions/rotations of objects in separate threads right? All has to be done after the jobs have returned the values to the main thread?

[velvet panther]

Yes, this is the main problem with Jobs, that it's really only suited for things that take more than a couple of frames, because of this limitation, and the setup costs of creating and distributing the Jobs, which is a bit much to be doing in a single frame cycle AND get the Jobs benefit from, and to then deal with the changed data.

[heady vortex]

would it make more sense to just keep a couple worker threads alive for the game duration then?

and just pass data to/from them?

[velvet panther]

If you're into threading, and know what you're doing, you'll likely get much bigger gains from doing it yourself than with Jobs, as Jobs is all about "ease of threading" rather than pure performance.

I'm a complete newb to threading and stackalloc, and my initial works are far faster than Jobs and even faster than Burst and Jobs.

[heady vortex]

and last question. Can I use burst and IL2CPP and still see performance gains with each other together?

[velvet panther]

I think so, this seems to be the big claim around Burst, that it's IL2CPP + IL2CPP with intrinsics focus!

[heady vortex]

👍 👍 👍

[velvet panther]

Of the three (Burst, Jobs and ECS) my gut feeling is that Burst is the most interesting, and the most educational (has been for me) and provides the most significant benefits with the least efforts. And understanding it and using it is portable to Jobs and ECS, if/when you may need either of them.

Having said, that I'm also tempted to suggest checking out using stackalloc and unsafe code within Unity, as it FLYS!!!

[heady vortex]

Any tips on what parts of code to use burst on? My code is heavily object oriented

[velvet panther]

My brain is heavily anti-OOP, so I've had an absolute blast with Burst and stackalloc and unsafe. It's the first time I've really enjoyed using Unity, so I'm probably the wrong person to ask about OO to Burst benefits. @wild onyx is probably able to answer this far better than I.

[heady vortex]

👍

[velvet panther]

two things that I think you have to take a look at, are the NativeArray containers and the low level capabilities you get "out of the box" as they provide a great backbone for doing all the Burst stuff over your data, and automatically mean what you make is likely close to being ready to Jobify if you ever need to spread it out that way, plus they're all thread safe containers if you're making your own threads... hang on... link incoming...

Jump up and down through the (admittedly light) docs from here, then use Rider's decompile to see the details: https://docs.unity3d.com/ScriptReference/Unity.Collections.LowLevel.Unsafe.NativeArrayUnsafeUtility.html

Also, JacksonDunstan, if you've got a day or two for reading, covers HUGE amounts of insight into both Burst and Jobs: https://www.jacksondunstan.com/articles/tag/burst

[velvet panther]

start "bottom up" - https://www.jacksondunstan.com/articles/5211

[wild onyx]

since this is in #archived-dots, i'd do some napkin math at the beginning to see if i even need to worry about certain parts of my project becoming a performance bottleneck. particularly in regard to neworking i'd not worry about transport level stuff until i have to... and before writing my own transport i'd look very very hard at stuff others have done, that has been tested in multiple projects and multiple scales as getting that stuff right takes time. Also many problems, performance-wise can be solved by custom messages, custom serialization and compression via bit-stuffing and delta...

performance wise i'd first try to get there with SIMD before using fine-grained parallelism... to parallelize network messaging it must be done in continuously running background threads with a very keen eye always on overhead caused by context switching, latency, cache wipes ... might even behave differently on different CPU/OS... also sync primitives can have unexpected behaviour...

[heady vortex]

@wild onyx I've already written most of the netcode for my game. I'm simply trying to stuff as many players on to a single core as possible (serverside)

clientside however

when the client decodes the bytes into actual quaternions and vectors, it also applies a lerp to them, since the server is running at 30fps while the clients are running at 60+

I just feel like I could benefit from multithreading that part

[wild onyx]

maybe... .but your message pipe into that thread and out if it have to be designed carefully to actually get a benefit

[heady vortex]

like a circular buffer?

lock-free?

[wild onyx]

yes, lock-free

maybe you can even use no synchronization at all... all depends on what you expect from the data, if glitches can be compensated elegantly

since its your project and not a generic solution you can probably get away with a lot of trickery

[velvet panther]

My gut feeling is that this: " the client decodes the bytes into actual quaternions and vectors, it also applies a lerp to them, since the server is running at 30fps while the clients are running at 60+" is exactly the kind of workload that will benefit hugely from stackalloc and SIMD considerations in how you structure the stacks you make for the incoming data, and the little results stack you pop the computed stuff onto after the SIMD magic before you apply it to your game objects.

[wild onyx]

would expect 10X over non-SIMD/parallel on a 8-core cpu

mind though that much of what you gain from cache & pipeline optimization (DOTS) gets wasted when data moves a lot between cores

but that again depends on how the CPU manages its cache

caveat: take that from someone who knows what DOTS is supposed to do but not how it does it in the current implementation

[heady vortex]

Again I'm new to multithreading, so what do you mean by this? Literally just a generic queue instead of currentqueue or something?

[wild onyx]

if you don't care if two threads write to the same memory location in undefined order, you don't need sync

[heady vortex]

ah thats what you mean by glitches

[wild onyx]

so you probably care for x += a * b but you wouldn't for x += a + b, given that you never reset x

big maybe though...

also, race-cars crash and explode... and race car engineering's aim that all of the car's components break down 1 m behind the finish line all at the same time

[velvet panther]

3asily the most programmery thing I'm going to read today!

[tribal pollen]

Thanks. So then on conversion (of a prefab Gameobject with mono) would I be able to store a reference to it's entity inside the mono?

[cerulean pulsar]

By default, the conversion systems destroy the GameObjects they process. You can override this behavior in different ways but we'd be in strange territory from my perspective. Also, you seem to not have any GameObjects for the conversion systems to process, because you said you don't have any subscenes and suggested that you don't have any GameObjects with ConvertToEntity on them, just a single GameObject with a setup script.

You could have the monobehaviour you're interested in create the entity you want a reference to, so when you instantiate that GameObject prefab via your setup script, the prefab instance will create the entity and will have the reference.

[karmic depot]

now that I think about it, depending on your use case, you could add your MonoBehaviour with AddHybridComponent. That would create a hidden GameObject with your script attached, for every entity. @tribal pollen and I think you can find the appropriate entity via CompanionLink component

[acoustic spire]

What technology for parallel programming Unity Jobs built on? OpenMP?

[robust scaffold]

I wish Jackson could get back to digging into Burst. I miss him

[velvet panther]

think he's gone to greener pastures.

[sinful cipher]

Its been a while since I really delved myself into everything DOTS related. There is as of yet no integration with physics right? Like, no way to interact with physics exclusively though DOTS/ECS?

[coarse turtle]

As in an alternative to Havok & Unity's DOTS physics?

Since those are the 2 libraries I'm aware of related to DOTS & Physics

[sinful cipher]

Uhm, not necessarily? I just want to do basic things such as apply forces without having to deal with UnityEngine.Object-based components such as the Rigidbody.

Because it becomes super complicated to make that fast and scaleable

[coarse turtle]

https://docs.unity3d.com/Packages/com.unity.physics@0.6/manual/index.html Might be worth browsing this and seeing if it fits your needs

I don't use it since I'm more in 2D land atm, but I know a few folks here use it

[sinful cipher]

Any idea when com.unity.physics 0.6 was released? I'm searching for it but maybe you already know

[coarse turtle]

Jan 2021

[sinful cipher]

Oh yea I see it now

thanks.

About as old as the latest entities package then

[sinful cipher]

Anyone here got any experience with Unity DOTS/ECS physics? Is it any good so far and worth investing time in?

[haughty rampart]

it's pretty good already.

from all the entities related packages probably the one with the most worked on features yet

[tight blade]

so, I have a question, and I'm just looking for a broad strokes answer, because I'm fairly sure this is a major category of program design that can't be answered simply

so far I've just been designing systems that are supposed to be always doing something - updating derived values every frame. but now I want to make a system that executes a single set of instructions in response to infrequently met conditions. like a button press.

I'm fairly certain that what I'm talking about here might be described as an "event", but I'm frightened to use that word, because I'm scared the answer is "yeah, you have to homeroll your own concept of event based programming and bolt it onto the ECS framework"

I suppose the triggered collision jobs are an example of event that already exists in ECS

so, I guess my terrible, overly broad question is... what are my options when it comes to running systems that dont run every frame, other than having those systems poll for those conditions every frame? aka, what are my options for responding to events

[tight blade]

hey, when you're digging into physics, feel free to @ me with questions. I'm not an expert in all of the niche use case things you might run into, but I've been working with it for a while

I can at least try to help with some of the initial gotchas

[sinful cipher]

That sounds great. Thanks alot!

[left oak]

@tight blade, have you looked at tertle's incredible event system? https://forum.unity.com/threads/event-system.779711/

[digital kestrel]

https://www.youtube.com/watch?v=6LzcXPIWUbc&list=PLAOytOz0HZbLaWhVrGEge5_6dNCAzGFYH&index=2

SIGGRAPH talk on hybrid DOTS

haven't watched it yet but it's on yt now if you guys wanna watch

[velvet panther]

is Hybrid what we get when they realise the "templates" of HDRP and URP are failed initiatives?

[velvet panther]

Also, check out DreamLatios work on a "physics" system. Merely reading this will give insights, I think: https://github.com/Dreaming381/Latios-Framework/blob/master/Documentation~/Psyshock Physics/README.md

Generally speaking, the failure to get a DOTS/ECS integrated physics system, particle system and audio system anywhere near a full fledged alpha state, and locking down of the general commentary on rendering, shader and lighting problems with DOTS/ECS, give the most accurate insight into where things are at. Project Tiny could have been, and now should be, the evolving surface for experimenting with ECS, in a separate editor. Especially now that it's clear WebGL Unity Monobehaviour Export/Builds can be made reasonably sized and performant. Attempting to grow leaves with branches before a root system had been idealised has meant that all these systems are now in stasis, and will need to be rewritten and/or re-amalgamated with whatever comes of DOTS/ECS in 2022, and only then can an alpha process for these incredibly significant systems begin in earnest.

IOW, we are in 2018, all over again, as far as DOTS/ECS is concerned.

The good news? Burst and Jobs. And soon... Span<T>

[golden ridge]

I'd just like to check for support of DOTS/ECS in the 2021.2 version of the Unity editor.

[gusty comet]

hi, I would like to do my end of degree project about high performance computing in unity, should I use dots/ecs or stick to C# jobs only? Because of the last messages I have my doubts

[digital kestrel]

your code will generally perform quicker with ECS

[deft haven]

Is it Possible to create a function in the blob asset script to Create and Return an Entity Archetype?

[safe lintel]

@golden ridge 2020 LTS is really the only officially supported version, though you may have success using 2021.2, if you have any issues you will just be referred to 2020

@gusty comet well its definitely incomplete, you should open up the sample projects and see for yourself if it suits your needs. i personally would recommend holding off now only because unity is not releasing updates for the time being, not necessarily because of the incomplete factor.

[tight blade]

nice, thanks for the reference! I only use physics systems as lightly as I can anyway, but I love love love having the physics systems around to lean on for prototyping and stealing library code

almost everything I know about ECS (and quaternion math) has been from looting the Unity.Physics package

so definitely glad to have another library to loot

and.. it looks like DreamingLatios engine is philosophically tailored for that type of usage pattern

nice

[gusty comet]

What should I do to fix that

arent bool comparisions single equal?

[sharp dragon]

There's no statement after your if

Just a ; which is an empty statement/does nothing as warning says

You can remove that whole line since the while loop breaks on the same condition already

[void ravine]

single equal sing is assignment not comparison

[gusty comet]

guess I should get some sleep, brain is super fried today. Good thing I aint programming anymore today.

[wise nacelle]

I'm guessing this question gets asked 200x a day, but I couldn't find the answer in the pins: is there a good course/tutorial on learning about dots? Preferably applying real world solutions. I get the abstract concept, but I'm having a hard time applying it. I found a course on udemy but it was last updated a year ago, did a lot change? Or should that be fine? https://www.udemy.com/course/unitydotsfundamentals

Also, there's something I've been wondering. I understand how you can achieve better performance with dots. But I don't understands if this also affects rendering. I have never made a game where my code was the bottleneck. It's always the graphics and how many objects I want to render at once. Does dots also tackle that? Allow me to have more visible objects?

[muted star]

I'm not experienced with DOTS but if graphics is your bottleneck DOTS is not going to help you much.
If you have thousands of objects doing stuff ECS can also be helpful to keep you code clean, other than that DOTS is mainly for performance.

[muted star]

Turbo Makes Games has up to date tutorials on DOTS and ECS

The documentation is also good

[wise nacelle]

All right, thanks. I'll watch those.

The thumbnails are a bit cringe though :p

I did read the docs, the abstract is clear. It's just very different to apply it to real world problems. I found some CodeMonkey videos that apply dots, pretty decent.

[muted star]

Don't judge a book by its cover ^^

[wise nacelle]

I'm not judging, just entertained 😄

[digital kestrel]

hey guys

how do we see the assembly code for a specific function

that was burst compiled

        public bool PassFrustumCulling(float3 point)
        {
            bool pass = true;
            var dot0 = math.dot(-Planes[0].normal, point);
            var dot1 = math.dot(-Planes[1].normal, point);
            var dot2 = math.dot(-Planes[2].normal, point);
            var dot3 = math.dot(-Planes[3].normal, point);
            var dot4 = math.dot(-Planes[4].normal, point);
            var dot5 = math.dot(-Planes[5].normal, point);

            pass &= dot0 < Planes[0].distance;
            pass &= dot1 < Planes[1].distance;
            pass &= dot2 < Planes[2].distance;
            pass &= dot3 < Planes[3].distance;
            pass &= dot4 < Planes[4].distance;
            pass &= dot5 < Planes[5].distance;
            return pass;
        }

i wanna see if that gets vectorized properly

[hollow sorrel]

@digital kestrel toolbar > jobs > burst > open inspector

[digital kestrel]

thanks

[golden ridge]

Should ISharedComponentData be used in projects? I saw someone on the forums stating to not use them at all cost.

[coarse turtle]

Depends on scale, if you have unique data that is shared between many entities, then you can use it instead of each entity having a copy of said unique data. If your entities are large and structurally change frequently, then you'll be paying for the cost of copying the data of entities between chunks

[golden ridge]

Something I am unsure about is about the existence of tags and systems that are for example called MakeEnemyMove that act only on thing with the tag enemy. I've heard that in ECS and Data Oriented design systems should have no distinction that if it's an enemy or not it only wants to know if it needs that system.

[left oak]

https://gametorrahod.com/everything-about-isharedcomponentdata/

[golden ridge]

Thanks you but do you also have any answers to my other question maybe?

[left oak]

https://gametorrahod.com/tag-component/

Though, note that in the future, tags might be a bit field header type of thing, but right now, they constitute a structural change

[golden ridge]

Can you explain it a bit more clearly?

[left oak]

Which part?

[golden ridge]

To reformuler my question, should tags exist in my code when using ECS.

*reformulate

[left oak]

Well, I tried linking you to a highly-informative article that would elucidate a number of uses for tags, but since you seem to want just a blanket statement: the only difference that makes a tag is that it is an empty component data; existential processing based on composition doesn't necessarily care what the values in a component are; so why not use tags?

[golden ridge]

Ok sorry I did not see the link you sent me. My phone didn't preview it.

[left oak]

Np, that author has a lot of articles (some technically outdated, but still principally sound) that might help guide you on understanding Unity's ECS

[robust scaffold]

Tags are very important as they allow for rapid "alignment" of entities for use in Entities.ForEach. Sure, they dont have any data within themselves but by fragmenting a chunk deliberately allows for far more fluid data processing.

For example, I have individual and bank entities that have a "wallet" component data. The difference between the two is that banks have a "Bank" empty component data added to it. In a first stage ForEach, I need to calculate the "interest" upon the wallet for every entity. That can be done with a simple query of all entities containing "wallet". A second stage now requires that all banks subtract a certain amount from their wallet, now I do a ForEach that also has a .WithAll<Bank>() parameter. Even if the "Bank" component has nothing in it, I can now easily differentiate between the two different types of entities.

And most importantly empty components is a chunk based component. It takes 0 memory per entity basis (but a few per chunk). That is important when you have millions of entities and need to manage memory usage precisely but dont want to waste a SharedComponent on it.

[hearty musk]

Can anyone recommend some “getting started with dots” resources?

[full epoch]

Code Monkey has good tutorials but they are somehow outdated dots changed a lot meanwhile but still i think it might be good starting point to learn basic stuff however as i said its outdated a bit - one newer is unity ecs roll a ball dots tutorial 2020 - by turbo makes games - you can find these tutorials on yt - also very good is site - gametorrahod dot com - but havent checked it lately but i think it's up to date - probably there are other sites that someone else will point to - edit : also manual of latest entities version is extremely helpful once you learn the basics also this discord channel and also dont forget unity dots forum where u can find a lot of solved problems explanations etc

[viral sonnet]

Does the ref work for anyone when using ElemenAt from a NativeList? I can't manage to write any values back to the struct but the return value of ElementAt is a ref, so, not sure why it's not working.

yeah, i dunno, ElementAt returns: return ref UnsafeUtility.ArrayElementAsRef<T>(m_ListData->Ptr, index); - though it's not working, so in that regard it's a useless method. huh

[viral sonnet]

ok, got it to work, code was wrong. this is how you correctly get it: ref var ele = ref nList.ElementAt(0);

[digital kestrel]

not that many imo. Code Monkey is good. turbo squid is pretty good. Personally, I looked at ECSSamples on github, read the documentation, and looked through Unity Dots forum

do you guys have both game object colliders and physics shapes on your game objects?

where the GO colliders apply only to the player character

[carmine dagger]

Hey im trying to implement a reward system (gold, xp, etc) before destroying an entity that has been tagged for death. I was able to get something working with a GameManager/Monobehaviours the downside being it had to be run .WithoutBurst(). Is there a way to implement something similar and retain the benefits of ECS? I tried updating another entity but couldnt figure out how to set component data of a different entity.
Im new to DOTS and ECS. Apologies if this is pretty basic. I tried searching here and google first along with a few youtube and Udemy videos which all seemed to use the monobehaviour workaround.

[viral sonnet]

@carmine dagger What's your actual question? Getting the entity with the tag component? Setting data on another?

[carmine dagger]

Setting data to another. I want the entity to be destroyed, and increment a value on another entity.

[viral sonnet]

There are 2 ways of setting. Linear or with random access. If you use a ForEach with your tag and a ref you can write to this data. Otherwise, ComponentDataFromEntity can lookup and write entity data.

The basic samples should get you started, in the beginning, it's easiest to utilize the Entities.ForEach

[carmine dagger]

Ill check out the ComponentDataFromEntity. The ForEach I couldnt get working. Tried adding a code snip but it was removed.

[viral sonnet]

removed?

[carmine dagger]

More specifically The ForEach is great when finding the things I want to destroy and cleaning them up. I couldnt get it to change the value of another entity.

Yeah. It flashes for a moment and gets deleted.

[viral sonnet]

not sure what that means 🙂 anyway, if you're in a ForEach and want to set other entities, CDFE as previously mentioned is the way to go

[carmine dagger]

This.

Its like being autodeleted. Either way. Appreciate the advice. Going to check out CDFE and see if I can get that working.

[inland depot]

Strange question but does anyone know whats happening with the ECS worlds?

I've been messing around with them and I had a cool idea about how to split up some work across worlds so that a simulation world can do some thinking while the render world just shows the player some stuff and allows for camera control while it's waiting for the simulation world to do it's job

but then I had a proof of concept working and noticed that it all runs sequentially on the main thread whereas I was hoping that each world would get a main thread and I could then schedule the cross world communication on my own 😦

[frosty siren]

Hi all! Should i add [BurstCompile] to my IJobEntityBatchWithIndex if i use it inside SystemBase in 0.17 ?

[left oak]

I have no idea how you're implementing this, but it doesn't sound like you're utilizing jobs in any meaningful way, because if you were, you wouldn't have everything running on the main thread (here's an interesting article on how to minimize main thread blocking when preparing jobs: https://gametorrahod.com/minimum-main-thread-block-with-out-jobhandle-overload/) ... The only time you need to come back to the main thread is during synchronization of your worlds

[torn spruce]

im having trouble installing entities

i installed from package manager using a git url, and the namespace is not showing up inside code

how on earth do you set dots up

[half jay]

what project settings should i use for app on android/iOS with ecs 0.16 and burst 1.3.9. Should i set ill2cpp or mono and .net4 or 2.0 ?

[cerulean pulsar]

I install DOTS packages via MyProject/Packages/manifest.json. Add "com.unity.entities": "0.17.0-preview.42", for example

[cerulean pulsar]

If the enemy moves in a special way, it could be important for the moving system to distinguish between enemies and non-enemies. I wouldn't consider movement speed a special type of movement, though, since all movable objects need a speed. You might have a Movable component with a MovementSpeed field for that simple case, rather than a tag

[inland depot]

What I mean with sequential is that the simulation updates in the same thread as the render world. So by default I have really long wait in my Simulation world while all the systems run, but what I want is for the systems to run in parallel with the other world which might update 30-40 times per simulation world tick

[left oak]

Did you actually make your own custom-bootsrapped simulation world, or are you just talking about the simulation system group, cause what you're describing shouldn't really be happening. You shouldn't be doing so much main thread work in one world that you're starving the other one. You should be able to offload most of your jobs to worker threads, and only come back to the main thread for synchronization and dispatching

You know, everything here sits atop Unity's underlying player loop. There's some stuff you can do to work around that, like calling update yourself at your own desired frequencies, but it's quite the rabbit hole

[muted star]

All jobs need the [BurstCompile] decorator as far as I'm aware of

[inland depot]

Custom boostrap with a custom IFixedStepManager implementation to manage the communication timing between worlds. So from my point of view the EntityManagers will never actually collide. The only thing that needs to be co-ordinated between them is the job scheduling but in an ideal world I would just give the render world 3-4 threads which take preference over simulation world jobs... cause for the most part The simulation world has few thousand items to go through but that could be split into jobs that take roughly 2-3ms each so that the render world can jump in and take resources

Actually triggering update is simple for me because I don't really care about the delta time it takes the simulation world, the render world just needs to stay responsive to movement when the simulation world has lots of stuff going on that makes it take long

I also don't really care about syncing up perfectly as long as the simulation world is essentially read-only when the updateManager tells the render world it's clear to read. Once all the transfers are done the render world will happily tell the simulation world todo another step so the time it takes the render world to do the transfer is essentially lost but that's an overhead that I can happily live with 🙂

But that whole thing doesn't really work if the simulation world is just scheduled for an update before/after the render world on the main thread 😦 as that just blocks it rather than being parallel. I really want a way to tell unity to stop managing the update cycle and just release the Simulation world and just give it a pseudo thread that acts as the main thread for that world specifically.

I feel like worlds are is just fighting the safety system though, at least the way it currently works 😦. It's not really a different world if it doesn't actually run separate to the main world, You can't really spin up a world to take the performance hit of a long calculation like you can with a job that just lives for a few frames 😦

[left oak]

Hmm... I think it's technically possible to call your updates entirely manually, but yeah, inter-world communication require coordination within the main thread. I'm a little surprised the transfer takes so long that it hangs your progress. If time is your limiter rather than memory, you could consider like a buffer world?

[inland depot]

It doesn't hang my progress yet, but if it does it's not an issue

[left oak]

Also, your simulation world can work during the transfer to your rendering world, but it cannot make structural changes safely

[inland depot]

My issue is that running the simulation world will eventually be on the order of Seconds-Per-Frame while the gameplay is fine with time-dilation. But I can't run them in parallel

At least I haven't figured out how yet

[left oak]

hmm, this might be elementary for you at this point, but there might be some hints here: https://gametorrahod.com/world-system-groups-update-order-and-the-player-loop/

[inland depot]

Currently my simulation world doesn't update until the render world moves it's data around (which is fine), so there aren't any data issues but some of the Simulation system have sync points which I don't see why the render world has to stop to wait for

Sure there is job scheduling stuff which might be more complicated but no data from sim world affect render world cause they are separate

Hence why I'd like to have the sim world have it's own 'Main thread' so that render world can happily keep going :/

[left oak]

I didn't think one world's sync point stopped all other worlds' jobs

[inland depot]

Well they all get scheduled on the main thread so you need to get all the stuff running from Sim world which means that Render world doesn't get it's update till it's all done

[robust scaffold]

Doesnt alternate worlds run on job threads?

as long as they dont have a presentation group?

[inland depot]

At the moment there is some stuff that I can't run from an Entities.ForEach so that all has to be done manually which causes a lockup, I'll need to have a another play with those at some point but either way scheduling should be done separately too (ideally)

but I can see why it's not cause it would complicate everything only for some edge cases like this

[robust scaffold]

Wait no, they're all stacked together in the player loop

[inland depot]

yeah :/

[robust scaffold]

If you're not already, you need a custom bootstrap solution. Something akin to the fixed step implementation

[inland depot]

See here

[robust scaffold]

except the player loop will instead access a public "end" dependency from this new world. Check for completion, if not then skip updating that entire world and continue on

Actually accessing the job scheduler I dont believe is possible. That's black box C++ code

so no "priority" scheduling sadly

The Render system already has their own specially allocated job thread, no need to worry about jobs locking the thread.

However, I do believe there is an issue about Entity jobs specifically lasting longer than 1 frame

The problem lies with the ComponentTypeHandles being invalidated at the end of every frame

[inland depot]

hence why I need an entire world for it which would in theory have it's frames last longer than the other worlds

[robust scaffold]

Normal Bursted Jobs can last however long they need because NativeContainers can be declared TempJob (4 frame lifetime) or Persistent (unlimited frame lifetime). ComponentTypeHandles are declared Temp for automatic memory disposal and thus cant last longer than a frame

So Entity.ForEach must be completed within a frame

[inland depot]

I have a feeling that I'm gonna have to go outside of Unity to keep playing with this little idea of mine at this point 😦

[robust scaffold]

The CTO's offer solution is somehow manually segment the chunk iteration, divide chunk count and then iterate over a number of chunk indices. Early return those outside the current frame's index values.

[inland depot]

Doing this using the job system is going to suck, I'd rather use ECS systems for the automatic schedule handling

[robust scaffold]

Networking and additional worlds at this point straight up dont work. They're half baked ideas casually attached to unity DOTS by ducktape and hope

If you want to remain in the Unity ecosystem but do something like you're describing, you'll need to implement your own ECS from the ground up

juggle native-arrays and pointer operations manually

[inland depot]

Unity provided networking has never really worked and worlds will probably get implemented after core ECS goes production ready at this rate 😦

I'd just give Bevy a go and see how their worlds stack up :/

[robust scaffold]

Yea, worlds are... intended to be used like what you're describing. Functionally independent system groups that only need to come together when scheduling.

But in it's current forgotten implementation, it doesnt work

Unity really needs to implement a "shared - chunk" component data

Shared blittable component data that is readonly-accessable within a threaded job

[inland depot]

Nah, customisable dedicated threads so that I can make a thread play the player update cycle for my simulation world without it moaning

maybe since 2021.2 is out there will be an ECS update soon

[robust scaffold]

At that point, you'll need a custom engine

Bevy seems decent. The documentation is a bit sparse but it's not like Unity is any better.

I just cant abandon Burst. It's so fucking good. I'm running off 2.5 million entities with random and dynamic access and burst can make that functional

[inland depot]

let me see if I can find something that might give some insight on that

https://github.com/nxrighthere/BurstBenchmarks

It's likely that the main reason it's so good is because it's good compared to monoC# or IL2CPP

but yeah it's a good tool to have

[robust scaffold]

Hrm, true. I'm not a comp sci or even a decent programmer. burst does all the auto-vectorization for me

and also the graphics engine. Yea fucking right can I make one of those myself

[inland depot]

some languages are better than others but it mostly depends on the compiler your using and if it does vectorisation/SIMD for you automatically

[robust scaffold]

I have 3 gameobjects actually rendered. A globe with a lot of fancy post processing. The entities thankfully never need to be rendered

[stone osprey]

Just out of curiosity, is there any ecs faster than unitys ?

Well there arent that many ecs for c#... Entitas is not that fast i guess... But theres still default.ECS and liteecs. Can those be faster than unity ? What does this depend on ?

[inland depot]

Faster how?
Faster startup, Yes there is.
Faster queries, probably.
Faster runtime will depend on what the systems are doing and the optimisation the language provides for the platform your working on...

[inland depot]

Also parallelism is a factor as some ECS libraries will be parallel without locking, others will update each entity in parallel (like unity) while others will just run systems in parallel...

[stone osprey]

Just faster in general, like faster iteration, queries or whatever :)

I just thought unity does some c++ magic under the hood which makes it just faster than any c# ecs lib out there

[inland depot]

Sure it has burst which will generate machine code from C# which is unique but other languages which make machine code in the first place will produce similaraly optimised code in a lot of cases.

I do think that burst will be a little better in the long run if your targeting consoles cause you don't have to make as many assumptions which the capability of the hardware... but you can also do those optimisations in other languages/engines it's just going to be more manual

[sinful cipher]

In Unity DOTS Physics:
If I author a composite collider object (say a cube + a sphere) and have it convert to entities at runtime, will the resulting composite collider be considered 'final'? As in: I can no longer move the sphere around in relation to the cube within the composite object?

[sinful cipher]

@tight blade Hey, perhaps you happen to know something about this

(I suppose I could just try this out tomorrow. It just crossed my mind as I observed that all PhysicsShape components are stripped from child colliders and merged into a composite collider attached to the parent entity.)

[gilded palm]

hi, i have problems with capsulecastCommand, i use the same params for both commands and normal call. While the normal call return hits correctly, i always get invalid RayCastHit with Commands!

[stiff skiff]

Quick question, if I store an Entities.ForEach query with a WithStoreEntityQueryInField. Can I set a filter on this query?

I know I can do it through the Entities API, but the filter needs to be set conditionally, so doing this through the stored query sounds best

[left oak]

You can set a change filter or a shared component filter in Entities.ForEach, is that what you mean?

[haughty rampart]

There are definitely other ECS frameworks, but Unity's will definitely be a very highly optimized and efficient one. Probably the best goto for gamedev. Lots of this is because of the burst compiler, but no, they do not do any c++ magic under the hood. Burst will even generate faster code than c++ code in lot's of cases. There's a small blog about this too if you want to read more on it if you are interested. https://lucasmeijer.com/posts/cpp_unity/

[stiff skiff]

Sorry let me clarify. I have an Entities.ForEach, that sometimes needs a change filter active, and sometimes doesn't

so instead of writing the whole thing twice, with and without, I want to know if I can use the WithStoreEntityQueryInField and set the filter conditionally on that

[left oak]

Here's some info about what you get from an Entities.ForEach stored query that I found in the docs: "The EntityQuery is created in OnCreate. This method gives a copy of that query that can be used at any time (even before the Entities.ForEach is invoked). Also, this EntityQuery does not have any of the filters that the Entities.ForEach invocation sets up." @stiff skiff

[sinful cipher]

good post btw

[stiff skiff]

Super, got it all to work

[sinful cipher]

Despite reading numerous descriptions of what a 'blob' is, I still don't quite understand it.
So far it appears that the basic idea is that you have a pointer to a much more complex structure (e.g. a collider mesh) that you'd rather not try to cram into a (shared) entity component?

[haughty rampart]

a blob is just an immutable piece of memory where all your complex (object-)information is stored.

[sinful cipher]

@haughty rampart Any idea where the name 'blob' came from? Just wondering. Maybe there was a particular thought behind it, and maybe knowing it might help it all click in my head.

Although if I had to guess: just a massive 'blob' of data or something

[haughty rampart]

pretty much. just a thing without any real shape, which still has all the (internal)properties you need XD i guess

[sinful cipher]

I see.
So here we have the Collider. And so it seems that it tells you "the data starts at pointer and ends after this size". And then all you get is the Type which informs you how to interpret this data, right?

[haughty rampart]

about right

[sinful cipher]

Okay. Thanks

Something physics/math related:
I often encounter the word 'Jacobian' in Unity DOTS physics. Is this some kind of integration method like Verlet, Euler & RK4?

[haughty rampart]

short: jacobian: solving a constraint matrix by linear approximation.

[sinful cipher]

Lol, luckily I just barely have enough math knowledge to even imagine what that's supposed to mean.

How many input constraints are we talking about? Arbitray number, or just two? Or is this a complete nonsense question I'm asking in this context?

Other than that, I have to say.. Its really cool all of this is just exposed to us like that in the DOTS physics library.

[haughty rampart]

yeah, more core parts of unity will also be rewritten in HPC#, so that might expose a lot more of unity itself too which would be great

[haughty rampart]

i'm not too deep into these physics myself, but my guess is only for 1

at least the calculations are

[sinful cipher]

Its funny you can just dig around and stumble upon the place where they do the basic 'position += velocity * dt' for everything.

[sinful cipher]

Oh right. I meant 'two' as in 1 joint between 2 bodies. So in that case my hunch seemed to be right.

[torn spruce]

where are the systems in the unity scene

im so confused

like where do they actually go in the scene

[coarse turtle]

they're not bound to a scene

systems will live in worlds effectively

[torn spruce]

just by the script existing in the project?

[coarse turtle]

in a way yes

because unity uses reflection (iirc before the first scene loads) and construct the systems from the types which exist in the assemblies

[torn spruce]

ty!

[deft haven]

I might have missed it, but I've read the Rules there was no mention that I can't post a Link, so here it goes... here a link on a my Forum post asking for help about how to create a "X" Definition Database kinda of thing in ECS.

https://forum.unity.com/threads/how-to-create-a-x-definition-database-in-ecs.1192864/

[haughty rampart]

what's your question?

[deft haven]

@haughty rampart how to create some kind of collections of "x" definition but for ECS, I use polymorphism in my Scriptable Objects

Ex: Item can be Equipment, Consumable, Material, Etc.
Abilities can be Passive or Active, that either have a Cooldown or Resource Cost. Or Both.

[haughty rampart]

that is generally not how you should think when working with ecs. your approach is incorrect

just create an entity with all the required data components you need. there's your item

[stone osprey]

Just out of curiosity... same archetypes are stored in the same chunk. Right ? So actually, a entity Player, Transform, Rotation is stored next to all other same entities. So if we wanna iterate over them, unity searches that archetype and iterates over those arrays.

But what happens if we wanna iterate over Transform, Rotation only ? They dont have their own archetype, because they are part of Player, Transform, Rotation. What does unity do in that case to get all Transform, Rotations from all archetypes ?

[haughty rampart]

first, they aren't arrays.
second, all componentdata of same type is grouped together

[stone osprey]

What no arrays ? And how does that look like ?

[haughty rampart]

it's just continuous memory

[stone osprey]

So lets say we have an archetype Player, Transform... all entities of that archetype would be simply stored in an block of memory like this ?
Player, Transform, Player, Transform... ?

[amber flicker]

This post previously helped me get a better understanding - perhaps it will help here - https://forum.unity.com/threads/ecs-memory-layout.532028/

[stone osprey]

Thanks ! 🙂 One thing i dont understand is... if everything is nicely sorted in arrays. How do we iterate over a combination of structs, which do not have their own archetypes ? An example

Player, Transform, Rotation, Mob, Transform, Rotation, Building, Transform, Rotation
Iterating over one of those archetypes is pretty simple, we search that archetype and start iterating.

But what happens if we simply wanna iterate over Transform, Rotation ? This one does not have its own archetype... wouldnt that slow down the speed ? What would unity do ?

[amber flicker]

Did you read down a bit in the post? e.g. does this not help answer your q?

[stone osprey]

Yes... but i dont really understand it ^^

Oh wait... probably now. So it simply looks into the matched archetypes and fetches the wanted components from there ?

[amber flicker]

it doesn't fetch - I think of it as just making an array of pointers to the memory. For any chunk there's a constant stride that can be used. Sorry - on phone - keep reading around it I think will help.

[stone osprey]

Alright thanks ^^

[haughty rampart]

pretty much

[stone osprey]

So unitys archetypes do use structs of arrays, right ? So the archetype looks like this : { Player player[], Transform transform[], Rotation rot[]... } ?

[feral minnow]

how to run unity DOTS sample project (https://github.com/Unity-Technologies/DOTSSample) on Unity 2020.3 LTS? I have these errors:

[sinful cipher]

Does unity have an official article detailing/explaining Blob Assets? (not talking about the API reference/docs here)

although this one I found is decent: https://coffeebraingames.wordpress.com/2020/11/29/getting-started-with-blob-asset/

[sinful cipher]

Thats how I always imagined it to be.

@stone osprey Although I wonder if each archetype has its own Rotation array for example, or if there really is just one single Rotation array shared by all archetypes/chunks

[digital kestrel]

hey guys, do you have a system where you switch between copying gameobject transform to entity and then entity to game object

like have physics handles entity transform, but when player activates some kind of root motion animation, the game object takes control of the entity

[stone osprey]

Can unity implement disabling/enabling of components ?

Or will they ? If so... how would they probably solve this issue ? How would they implement it internally ?

[amber flicker]

it's been planned for over a year - who knows when it will drop but they did come to the conclusion it was necessary so unless that's changed, hopefully they'll deliver at some point. They plan to implement it just with simd backed bools. If you search on the forums you should be able to find a few posts about it.

[stone osprey]

SIMD backend tools ? That sounds... pretty lowlevel to me 😮

Thought they just gonna add a flag or whatever to each cmp

[amber flicker]

Exactly. By simd backed, I just meant that the filtering chunks will be fully simd'd (& bursted). It will still presumably slow everything down though so it might be why we haven't seen it yet. Joachim seemed confident about it a year ago (from memory).

https://forum.unity.com/threads/dots-skill-system-repo-available.894007/#post-5875804

May 2020... oof "internally this is done" - present tense

[stone osprey]

Omg thats some low level stuff i dont really understand 😄
So each component basically gets an additional bitflag/mask to mark them as disabled ? Im currently trying to implement such a mechanic into my serverside ecs which is also archetype based and just wonder how i could implement something like this.

[haughty rampart]

you can disable entities by adding the Disabled component

[digital kestrel]

if I have multiple entities adding damage components to the entity via command bufers

how do i ensure that

1. the entity has a damage component to begin with
2. the entity's damage component is stacking damage each time it gets added

Imo with command buffers, there's no way to query the value of the damage component until after the command buffer's sync point is run ... unless I just leave the damage component on the entity and reset the damage to 0

[haughty rampart]

You can't have multiple components of the same type on 1 entity

[digital kestrel]

does using dynamic buffer for damage component make sense

each system adds its own BufferElement for how much dmg it did that frame

[haughty rampart]

Why do you need multiple damago components for 1 entity?

[digital kestrel]

different sources of damage

[haughty rampart]

Why don't you just deal damage??

[digital kestrel]

idk

it made more sense to modify the health in a single system and have other systems add damage components than to have a bunch of systems manage the health directly

[haughty rampart]

That is not how ecs works and this is not how you should think about ecs

[digital kestrel]

oh sorry, i got that idea from a Blizz dev who gave a gdc talk on ECS architecture, but if you'd like to elaborate on why, I'm all ears

[wanton plaza]

could anyone explain how to properly convert a subscene manually?
i require determinism, but would like the subscene conversion workflow in the editor (then load that at runtime)
currently, i have to manually step the world a few tics just to get the subscenes loaded in, but i should be able to load them in immediatly.
I was pointed to serializeutilityhybrid for serializing subscenes, but fail to understand how to properly get the entityrepresentation of the subscene.

so, given a scene with autoring objects, how would i obtain a scene in entityrepresentation (that i can serialize in the editor and load at runtime?)
i couldnt find this in the docs:(

[sinful cipher]

Is it possible to take a pointer to a managed object (e.g. GameObject) into a Burst job?
(Not to actually use the GameObject in the Burst job of course. I intend to have a pool of available GameObject(-pointers) that I can assign to various Entities within said Burst job.)

Or is this really not safe to do?

[shell berry]

You can try it and see if the Burst compiler spits out errors.

It's really not recommended. What you can do is copy the managed data to unmanaged one (either by entities or structs in NativeArray). Then use the unmanaged copy to Bursted jobs.

[sinful cipher]

Hmm, alright, thanks.

[sinful cipher]

Yeah so far I already have something like that. But it always involves a lot of additional collections and tables just to be able to know where to take the data from and where to put the data back when the Burst job has finished. So I hoped I could just pass around a pointer and be done with it.

[ocean sparrow]

Hey, I assume ECS is not really production-ready and should be avoided?

[haughty rampart]

It is encouraged to test and play around with ecs. It is not recommended to use it for production. Some studios already used it to publish games though.

[torn spruce]

what do the GetSingleton() family of functions do

im having trouble finding documentation and how to use singletons in dots

[left oak]

It basically caches a query that consists only of the specified component and its associated entity, @duuke

[haughty rampart]

it also ensures that there will never be more than 1 entity that matches the query

it throws an error if more than 1 entity is found

[viral sonnet]

Wanna hear something funny? Dictionary is out-performing NativeHashMap. 100k loop -> 32.08ms vs 42.7ms - so much for a high performance container. lol I always wondered why my code with a hashmap was so slow. Even funnier, ComponentDataFromEntity outperforms both with 23ms

[zenith wyvern]

Try using NativeHashMap with burst

Native containers are known to (usually) be slower than managed containers in managed code. Burst is where the speed comes from

[viral sonnet]

let me try something to confirm but I don't think that's really burst related

burst is enabled anyway, i'll move to a burst method maybe it does something

[zenith wyvern]

If you're not using burst then theres no point in measuring it against a dictionary

Yes you need to use burst compiled methods to write to your container. I can all but guarantee you will see a dramatic difference

[viral sonnet]

Compiled as static burstcompile method. no changes. You sure you know what you're talking about? My understanding is Burst can't improve NativeContainers unless they can be vectorized in a loop, and my test can't be vectorized.

[zenith wyvern]

You said you're writing to a container in a loop, that is what my assumption is based off of. From my experience there should be a dramatic difference doing that with a dictionary vs a bursted loop with nhm

Now you have me second guessing myself though, I will test this when I get home

[viral sonnet]

no, i'm just reading. all data is pre-filled before the test

well, this is getting weird. i mean, it's just a test. usually i use nhm in burst compiled jobs but I'm not able to get the test method burst compiled without some errors being thrown. i usually never have the use for single burst methods. it's only showing up when it's static and when it is, it complains about nhm being managed without ref and with ref burst tells me to use a pointer. i'll try some more 🙂

[coarse turtle]

i tihnk the shitty performance with NativeHashMap is pretty much the jitted code not being optimal outside of burst 🤔

it might be generating a ton of instructions in mono ¯_(ツ)_/¯

but yea got curious with what you said on 100k and it's pretty much what you described 👀

[viral sonnet]

I scrapped the static method as I couldn't get it to work and wrote an IJob now with the same code. Now it's so fast I'm wondering if the code is running correctly haha - that's some really interesting performance characteristic. granted, the loop now probably gets heavily vectorized but that also the lookup itself is faster is pretty crazy. in nhm it's a lookup into a bucket and then iterating, guess this iteration vectorized is the big difference.

what I don't understand though is why the nhm isn't burst compiled from the get go then.

[coarse turtle]

dunno, think it might be how burst is still kind of separated from Unity in general so the compiler won't link the AOT generated code from the mono runtime.

[viral sonnet]

that must be it

[coarse turtle]

.net 7 will get NativeAOT so maybe there's some hope there, but afaik Unity will still stick w/ IL2CPP since it's more mature than NativeAOT

[viral sonnet]

sorry @zenith wyvern, you were correct 🙂 my test setup was garbage without utilising burst.

[zenith wyvern]

No worries, good to know I'm not crazy, hahah

[viral sonnet]

good thing I posted here, now I need to rewrite my other tests to burst too. basically I want to compare a few different methods of getting "database" items the fastest way possible. question was, use ecs with chunk access, CDFE or use hashmaps

[zenith wyvern]

Fwiw from my experience cdfe and hashmaps are pretty dang slow compared to a nice loop that burst can tear through

[viral sonnet]

yeah, that's for sure that that would be best. sadly, it's random access data that I can't change. I tried it the non-random access way but the overhead was massive because that meant i needed to either create new entities or write to a NativeStream and both cases were slower than just doing random access without all the writes

[zenith wyvern]

Yeah it sounds like you're on the right track with cdfe then. Also using a native container can introduce a lot of dependency headaches if you end up sharing it between systems too so I tend to avoid it if possible

If you're only ever reading from it I guess it wouldn't be an issue though

[viral sonnet]

yeah, the data is never changing

[zenith wyvern]

Have you looked at blob assets? I think someone on the forums made a dictionary blob. Not sure how it would compare to nhm

https://forum.unity.com/threads/hashmap-in-a-blob-asset.779000/ if you're interested

[viral sonnet]

thanks, i'll take a look

[haughty rampart]

that is correct. they'll be keeping an eye on nativeaot though

[viral sonnet]

the test is a little unfair for CDFE but my previous estimate was that one CDFE lookup is rougly the same as one nhm lookup. the test says now 1 nhm lookup = 1.25 CDFE lookup. i'm looking for a good solution that returns all comps from an entity. any ideas? the best I came up with is just building a struct with all the comps and throwing that in the nhm

the memory layout of ecs is not helping much for getting all comps from an entity because the comps are split up in arrays so no way of just reading the complete struct of icomps

[haughty rampart]

just get all the components you NEED, it's not made for getting ALL components without knowing what you actually need

[viral sonnet]

I need to get at least 5 base comps. I could merge them but there are the optional comps then which need a HasComponent and then the get which adds up. i'm looking into pre-building a blobasset right now

i need to reduce the lookups to exactly 1 instead of 1 ... n

the lookup is for RPG spell data

[torn spruce]

how do you guys serialize properties to get them into ecs world

[haughty rampart]

wdym? you want to set things in the editor?

[torn spruce]

how do i get a reference to a mesh

or sprite

or material

to use for example as a tile sprite

for my native array of tiles

[haughty rampart]

to get it from an entity, get the shared component

[torn spruce]

im not sure i understand

how would i geta serialized attribute on an entity in the first place?

and then how would i access the shared component of that specific entity?

[left oak]

please read the documentation: https://docs.unity3d.com/Packages/com.unity.entities@0.17/manual/index.html

Also an article on shared component data: https://gametorrahod.com/everything-about-isharedcomponentdata/

[valid hare]

I've made a editor time monobehaviour to test my pathfind. It does allocate a NativeArray on OnStart, but OnDestroy not always run.
What method should I put my disposal action?

[haughty rampart]

OnDestroy()

[pliant pike]

wouldn't OnStopRunning() be better 🤔

[haughty rampart]

actually yeah

[gusty comet]

Burst question: where does the code end up after compiling?
Regular script code goes into GameAssembly.dll. But Burst does native compilation from the start while IL2CPP runs later.
How do regular and burst code call each other?

[deft stump]

are there tweening libraries that uses jobs and burst? I know there's jtween, but it's really old now

[haughty rampart]

calling burst code from managed code comes down to PInvoke calls. so, depending on what you do, calling burst code from managed is somewhat slower

[haughty rampart]

none that i know of. also: Unity is / has been working on .net 5 / 6 support for many months now 🙂

[sinful cipher]

Should I be modifying Blob asset data after it was already created? Or should I treat it as pure read-only data?

[safe lintel]

@deft stump timboc was working on one I believe

[haughty rampart]

blobs should usually be immutable

[shell berry]

There was someone years ago that shared a tweening library using DOTS in the forums.

[amber flicker]

Re tweening...yea.. probably me 😬 . I got pretty far with it (complete authoring solution for reusing animations and tagging arbitrary game objects or entities, fully dots backend, arbitrarily nested tweens etc) but it's on pause. At the very least until Unity starts releasing packages again. This was the most recent time I posted about it: https://twitter.com/timboc_/status/1448982080617070592

Tim Watts (@timboc_)

About 200k cubes animating on the CPU @ 60fps just because https://t.co/PkzpI5Wg8M

▶ Play video

Twitter · Oct 15, 2021, 12:00 PM

Some surprisingly hard parts to this problem (in case anyone else is thinking about it) are:

• Being fast if you support a hierarchical/nested structure
• Being maximally parallel yet supporting e.g. GoTo(x) - especially if you support relative values for your tweens (I attempt to)
• Semantics - if you want to go fast, 'pause' doesn't pause immediately blocking on main thread - so if you type: tween.Pause(), tween.GoTo(4), tween.Play(), what should happen?
• When does an OnComplete fire?
• Designing an API - do you try and support ISB's? - when you want to do anything you have to pass ref systemState? Then what about when you're outside of an ISB? It means you need a tween to return completely separate representations depending what type of system it's used in? Static extension methods? etc etc. Refactor 5 of the API is currently where I'm paused.
  Then once you have all that structure in place, cry as 1 tween takes as long as 5000 tweens due to system & scheduling overhead 😆.
  I could go on forever.. already spent way too much time on it.

[last swan]

Heyo, is there a way, to change this layer in a system?

[pliant pike]

you can just get gameobject things same as you would in a monobehaviour

you just can't use them in full bursted jobs

[last swan]

I have tried to get the gameobject from the entity with GetComponentObject, but it returns null (with transform), or it says component is not added to entity (with gameobject). The picture shows the generated entity which's gameobject I try to get access to

[pliant pike]

I haven't tried to do it the way you have but personally I would keep the gameobject as a gameobject if you really want the layer

I'm not sure if the entity type has a layer 😕

[last swan]

Well the problem is, I instantiate an object from as addressable prefab, which already contains the ConvertToEntity - So I thought, I could just query them and change the layer in a system instead of iterating through the complete scene with all gameobjects and check for the SerializeableAddressablePrefab Component and then set the layer..

[pliant pike]

maybe you could make and add your own layer component 🤔

[last swan]

thanks for the input, haven't thought of managing this in an own component

[pliant pike]

no prob, enums or tags should suffice

[stiff skiff]

Hmm, .NET 6 got released today

And it shows unity as a platform?

[amber flicker]

@stiff skiff This is a pretty gold post on their progress and plans (though nothing dots specific): https://forum.unity.com/threads/unity-future-net-development-status.1092205/

[scenic oracle]

Anyone has infos on the current future of ECS?
Recently wanted to start an actual project with it, only to find it scrubbed from the usual package manager, with an official forum post saying they gave up updating it for a year on modern Unity versions, which is kinda problematic for me.
... any news since then?

[devout prairie]

Is there an early exit for Entities.ForEach ?

[frosty siren]

https://forum.unity.com/threads/is-dots-being-abandoned.1183621/

[frosty siren]

you mean return; ?

[devout prairie]

hehe yeah i guess so 🤔

[viral sonnet]

How can we make sure ECB.Playback uses burst? I read a bunch of times that it should work with burst but it's slow and doesn't show up green in the profiler so, I'm not sure if this is working

should we run Playback in an IJob or something?

[haughty rampart]

dots development will return at the end of the year or a bit later

entities package is currently only available for unity 2020 until further notice

[pliant pike]

I don't suppose anyone has any idea how I'd get info from a scene component, a GenerateAuthorComponent into a GameObjectConversionSystem

or maybe you can't because things don't exist at the same time 🤔

[viral sonnet]

Where do you want to get the data? Convert stage?

[pliant pike]

nevermind I just changed the generateauthoring to a monobehaviour and that seemed to work

[viral sonnet]

GenerateAuthoringComponent creates a hidden MB comp AFAIK. best to create a MB yourself

[pliant pike]

yeah I don't know its weird how it works

I can't get it when its a component like that in a Gameobjectconversionsystem probably because the component doesn't exist yet

[viral sonnet]

it is, I rarely use it over the 3-4 lines of code

yeah it's hidden and live under a componentName_Authoring.cs file. Not sure how it's handled under the hood

[haughty rampart]

it generates a simple monobehaviour under which just goes through the same conversion system. it's awesome for most components. but yeah, you can't customize the generated behaviour yourself

[scenic oracle]

@frosty siren
So the good new is that it's not abandoned, and the bad new is that 1.0 has no released date and will probably have drastic changes compared to current system?
... guess I will try another project in the meantime and wait.
Thanks for the link.

[frosty siren]

There are a lot of speculations on forum. But yes, there is no release date, even no speech event date, so the only we can depend on is the knowledge that dots will be incompatible with 2021 unity at least for the end of the year.
There is a little less silence on forum from devs, we can hope we'll get updates soon, but there is nothing except hope, so yes, if you gonna develop your project fast, i think it is better to avoid using Entities package.
Also i have to point that we know great masters on forum who have already released projects written with Entities and they say that it is more then possible to make a game with 0.17 version.

[safe lintel]

I’m not getting my hopes up they get back to public releases in any reasonable time frame. There’s no reason to be optimistic about their estimates and every reason based on the past, to expect longer(would love to be wrong though)

[frosty siren]

I agree

[viral sonnet]

I made a thread some days ago but it didn't answer the problem really. DynamicBuffers are painfully slow and I don't get why. Now I've rewritten the system to use blob data instead (which also has the same array data that was previously in the DynamicBuffers) and the timing got halved! I expected being a little slower or a little fast but not on that scale. The buffers are small enough to reside in a chunk. Replacing them with the blobdata made the chunk capacity much bigger though so that's some increase as well. I really don't get the performance characteristics.

I quickly had the thought, ah I've some write data but no, the write data is in a code path that's not used in what I'm testing right now.

and this is with 250k entity count. Maybe it's too much but I know the performance characteristics of 250k arrays and there it's not a problem

I've also not removed the dynamicbuffers yet, so they are still in memory. That means iterating over the blobdata is faster than iterating over the dynamicbuffers which makes no sense to me. Both should be pointers, but one is pointing at a random position, the blob, and one is in linear memory space

the only answer that would make sense is that the DynamicBuffers are actually randomly located in memory

[rotund token]

i've been watching the thread and i don't really get your results either

and it doesn't match our own testing of buffers so i'm not sure what's going on

and i don't really see why blob data would be any faster

[viral sonnet]

oh, hey tertle! yeah, eizenhorn said the same thing. something odd is going on :/

[rotund token]

unfortunately i just haven't had time to try replicate your tests

(this type of optimization is really interesting to me so i'll get to it at some point)

[viral sonnet]

i think it's best i make a small project to reproduce my problem. just to make sure everyones getting the same results or I'm doing something really stupid/easy to explain

the thread went a little off-topic and I never posted one

pains me to say, the more I write, the less of Entities i use

burst is really great though. it's really the kind of high performance code one would expect in 2021

[rotund token]

i agree if you're writing extremely high performant simulations

i find entities is the glue that keeps my implementations together

and lays out data in a way i can easily and quickly manipulate it for the specific simulation

[viral sonnet]

yeah, same here. it's not that i could completely ditch it, it's essential for lots of parts

funny thing I had yesterday. there was a race condition in a parallel system so I'm writing to a stream and then the stream is read and a nativehashmap is filled. that ran in parallel and performance was terrible. something like 10ms or something. Then I used a main-threaded IJob for it and the timings went down to <1ms.

NativeHashmap and NativeList seems terrible for ParallelWriting

anyway, to be expected with a little experience from anything that uses Interlocked. still always funny to see how multi-threading crashes and burns

and I kind of wish Unity removes either the Interlocked or the ParallelWriter altogether. It goes completely against the performance by default.

both are also terrible to use in parallel when you need to pre-allocate

[rotund token]

not sure how you would do a parallel write on a hashmap without some type of interlock

that said i have this extension method for parallel nativelist

            where T : unmanaged
        {
            idx = Interlocked.Add(ref nativeList.ListData->Length, length) - length;
            ptr = (T*)((byte*)nativeList.Ptr + (idx * UnsafeUtility.SizeOf<T>()));
        }```

that lets me allocate a chunk of memory at once if I know how many elements I'm going to write to avoid constant interlocks

then I can just directly write to the reserved memory of the list

much faster

[viral sonnet]

oh, interesting. so that avoids the interlocked waiting in AddNoResize?

[rotund token]

yes it's basically just AddRange except it doesn't add anything instead just reserving the memory for you and returning the ptr/length

so you can write direct

if you add 10 elements per thread, that's 10x less interlock calls

etc

[viral sonnet]

that's cool. thanks! that problem pretty much blocked me from using nativelists

say, i wanted to ask this some time ago since you have written a nativestream. would it make sense to add a ResumeForEachIndex? BeginForEachIndex clears the elements so the stream isn't really designed for adding an unsorted amount of indices

sadly it's the only nativecontainer i know which can add an array to indices that is also fast enough. can your trick be extended to a nativehashmap?

[rotund token]

it could probably be extended to native multi hashmap

assuming you were all using the same key

but it can't work for different keys, it needs to check the key is unique

[viral sonnet]

right, and I don't really know how much elements end up for an index

[rotund token]

and the only way to check this is really to lock other threads from writing while you do this

now a faster multi hashmap would be really beneficial to me

i have this Optimal Reciprocal Collision Avoidance implementation that can handle 200k entities in 6ms

but 3ms of this is is building a spatial map

funny enough, the fastest implementation i ended up finding was a spatial hashmap

it out performed units BVH in physics 4x

[viral sonnet]

whoa

[rotund token]

and while it was 10% slower than a native quadtree implementation for reading from aron granberg (a* pathfinding guy)

it was 2x faster for writing to it because the quadtree was single thread writing

if i could get a faster hashmap like structure i could really improve this

i just havent got around to looking in to write it

my only theory is do a stream like first pass without locks

[viral sonnet]

writing nativecontainers takes a lot of time

[rotund token]

then combine the different streams together on the key

maybe that would be faster, i'm not sure

[viral sonnet]

yeah i was thinking abusing the nativestream. you'd be restricted to the integer keys though but in entities you can get the entitychunkindex easily

[rotund token]

            this.Entities
                .WithAll<ORCANeighbours, ORCAAgent>().ForEach((int entityInQueryIndex) =>
                {
                    writer.Add(Hash(Quantized(agents[entityInQueryIndex].Position, quantizeStep, halfSize), quantizeWidth), entityInQueryIndex);
                })
                .WithName("Quantize")
                .WithReadOnly(agents)
                .ScheduleParallel();```

```        [MethodImpl(MethodImplOptions.AggressiveInlining)]
        private static int2 Quantized(float3 position, float step, int2 halfSize)
        {
            return new int2(math.floor((position.xz + halfSize) / step));
        }

        [MethodImpl(MethodImplOptions.AggressiveInlining)]
        private static int Hash(int2 quantized, int width)
        {
            return quantized.x + (quantized.y * width);
        }```

this is literally all it is

quantize the position as a int key

[viral sonnet]

yeah, really straight forward

[rotund token]

and finding nearest neighbour is just checking your quantized cell and the ones next to you

then just brute force checking all the candidates

but yeah, if i could improve that top job would be fantastic

but again, this is 200k entities which is basically the benchmark i stress test all my libraries on

but in reality i'm only looking at having 10k active agents in the world

which would let me run 20 really complex simulations!

[viral sonnet]

you go crazy like me with testing. i like that! 😄

i wouldn't catch anything with only 10k testing.

so, yeah, faster nhm would be great. maybe we come up with something

[rotund token]

well yeah it's hard to optimize on 0.1ms run times

that said i just spent 2 days at work optimizing away 0.3ms main thread code 😐

[viral sonnet]

hey that's a lot when you want to target 144fps

[rotund token]

i'd like to get our console to 30fps

😅

[viral sonnet]

hehe, well, slowly chipping away

[coarse turtle]

I can relate 👀

[robust scaffold]

imagine if there was a way to use a tag to [UpdateInWorld]....

[sinful cipher]

Is this a "valid" way of looking at chunk components: Useful for when you would like to use a shared components, but also want to use it inside of a job?

And is my understanding here correct: For all entities within a chunk, a single instance of a particular chunk component exists for all of them? (As in; the chunk component is not unnecessarily duplicated for each entity within the chunk.)

[coarse turtle]

on a system? Yea it'd be nice out of the box, but otherwise still doable w/ ICustomBootstrap

[robust scaffold]

It's really fucking annoying. I really wish Unity included a "static chunk" component, a component that could fracture entity chunks dynamically with blittable values that could also be accessable inside a job.

[robust scaffold]

Yes. The chunk component is a regular component located on a "header" entity that is unique per chunk.

[robust scaffold]

If you want to access a share component within a job, you need to use a chunk component and a main-thread job that "mirrors" the value of the shared component to the chunk component.

it is incredibly fucking annoying

[robust scaffold]

Unity and crashing using pointers + burst, name a more iconic duo...

also im using entities on 2022.1 so that may be an issue... nahhh

[robust scaffold]

5 hours later, I've done it. I've offloaded my entities authoring onto a job thread.

Did you know: you can stick (almost) anything into a IJob? From managed lists to strings to Newtonsoft's JsonConvert. I've been blinded so much by Burst that I forgot there exists normal C# out there.

[viral sonnet]

is there any performance difference to doing a main-thread loop then? any job without burst seems rather pointless to me

[viral sonnet]

i've a test system now that pits DynamicBuffer against NativeMultiHashmap. NHM is a little faster, 3ms against 2.7ms. There is 1.66ms overhead for nhms to get the entity array for the keys which I don't account for. in that sense it's not completely fair but I don't care because the entity array can be cached. This is quite the odd performance characteristic

i don't quite understand why DynamicBuffer is not faster when the access should be completely linear

NHM has quite a lot of needless iterations to find the correct key in the bucket + GetHashCode

[hollow sorrel]

you can use .AsNativeArray() on dynamicbuffers and it should be faster to iterate

[haughty rampart]

jobs still let you multithread things.

[viral sonnet]

@hollow sorrel I've tried this before as it was suggested the last time to me but it makes no difference.

DynamicBuffer vs. NativeMultiHashmap

if anyone is interested, feel free to try and report if you get the same results

more details: https://forum.unity.com/threads/dynamicbuffer-t-performance-doesnt-scale-what-to-do.1185427/#post-7645039

[ocean sparrow]

Does ECS as it is right now benefit most games, or only specific games?

[viral sonnet]

I'd say most games just because the data oriented design is a lot more structured. On that front I'm not even talking about the performance gain but maintainability, extensibility and more stable code without the spaghetti nonsense

[pliant pike]

yeah I prefer it just for that, the performance aspect is just a bonus to me

[robust scaffold]

I spin up about 2.5 million entities so loading takes a solid half minute or more. But shifting this loading onto a job thread, loading no longer blocks main thread graphics update so now I can include a spinny circle so prevent people from thinking the game crashed on loading.

[viral sonnet]

ah, pretty cool

[robust scaffold]

It's actually slower due to it now being in one thread instead of partially multithreaded (ExclusiveEntityTransaction can not be used in multiple threads) but having no blocking loading sequence is worth it.

[viral sonnet]

i guess ECB would also be not much of help?

[robust scaffold]

ECB can be used with exclusive entity transaction, theres an overload for the playback function for EET specifically. Problem is, there is no way to schedule that playback as an IJob since it must be main thread......... maybe. I need to look into that. My jobs are passing around a lot of managed data like strings so I dont think itll work in IJobParallel anyways.

The fact that there is an overload for EET playback implies that it can be used inside a job. Hrm

[viral sonnet]

ah ok, maybe ECB makes some code better but in that regard, hard to tell. I don't think you'd get more performance out of it. That's only happening when entity instantiation can be batched

if that's even your bottleneck 🙂

[robust scaffold]

I'm keeping an eye on your DB thread and honestly, I'm stumped too. Never really bothered to profile the performance of them but yea. It is odd that NMHM is winning out

[viral sonnet]

my previous trigger system wrote buffer elements to the entity. now I pre-allocate a NHM (entity 2 index) and a NativeList with ref ElementAt to set flags. it's sooo fast

thanks! yeah, hope we get to the bottom of it. there must be some kind of explanation

[robust scaffold]

You can cache ref element at into a list?

[viral sonnet]

not the ref directly but the list is pre-allocated

i think the ref could also be cached. lol, that would be something

[robust scaffold]

Can you paste the code here on what youre doing?

The only problem I see with using a NHM is that it cant be vectorized.

[viral sonnet]

true that. I could only post fragments, not sure if that makes more sense then

[robust scaffold]

Sure. I can maybe infer the context

[viral sonnet]

basically I'm writing a spellcast system for WoW/FFXIV like spells

[robust scaffold]

Never played either of those games but okay

[viral sonnet]

are any other RPG spells 🙂

the problem is many parts are not linear for processing. and I tried making it linear but that was overall much slower

as example, imagine a character has a trait that on every crititical hit he's taking he'll explode with an area of effect damage

[robust scaffold]

Hrm. Yea. I can see why that might be an issue.

Having to check every single entity for validity will be problematic

[viral sonnet]

right, it's a lot to process. I'm testing with 250k spellcasters that are casting every 1.5seconds. as they are aligned one frame has to process 250k spells and everything included. been chipping away for 2 weeks now to get this down. some major jumps in performance improvements. I started with 330ms and now I'm at 36ms

I would really like to bring this down more but now I'm at a point where improvements are not that drastic anymore. most systems are not that spiky anymore

[robust scaffold]

How many effects are there? And 36ms for data check only or do you also have graphics (positional / rotational) changes as well?

Because 36ms is horrible for only 0.25M entities.

If you have a hard ceiling on the number of spells, say 128 or even 256, you might be able to instead use bit flags and check via component. Now I dont know if bit operations are vectorized though.

[viral sonnet]

yeah, not that great still, i read the wrong numbers and I've some testing parts currently running. From memory it's 10ms for the checks if the spellcaster is even allowed to cast (range, resource, angles, etc...) and spell creation. and 12ms for spell calculation.

some parts are pretty heavy like getting all the target info, distance, angles, etc...

but yeah, i also think this should run faster 😄

[robust scaffold]

Hrm that seems better but that's roughly the amount of time I'm smashing 2.5 million entities together for. Doing financial calculations: interest payments, dividend payouts. Lots of relational mathematics that I've managed to largely vectorize.

[viral sonnet]

I think most code is not vectorized so that's really an issue

[robust scaffold]

In my experience, taking a small 0.5ms or even 1 ms cost to use dynamic buffers over faster more specialized Collections is well worth it if you manage to twist the Burst compiler to vectorize the operation.

Because that slashes computation time by multiple ms when accomplished.

[viral sonnet]

need to check at some point about the vectorization. the spell batch should run completely in parallel because it has no race conditions

[robust scaffold]

Maybe copy paste the spell cast check here and I'll see if vectorization is possible.

The main problem of course is the random access to obtain the target of the spell... that can not be vectorized.

[viral sonnet]

yeah, there's so much stuff in there that can't be vectorized

and in the first/second iteration I was breaking down the systems in parts but that didn't really work out because I didn't know what to with half the data and how to send it to other systems

[robust scaffold]

At that point, ya gotta think about completely overhauling the foundational design of the system to something that can be vectorized. Step away from traditional game design of spells and maybe come up with something new and better suited for DOTS.

[viral sonnet]

like a spell can be either single target/multiple targets. so you don't even know how much targets you end up with

[robust scaffold]

Yea. Hrm that is an issue.

[amber flicker]

In that case, wonder if you could cast multiple single spells instead? Just a thought.

[robust scaffold]

The main problem here is the relational aspect between the caster and target(s). I managed to step around this using chunk components containing the "caster" and entities within that chunk being the "target". That allows for vectorization as theres one random access and a vectorized data process over all the targets.

But the "caster" and "target" are basically set at loading so theres largely no movement of entities. With actual spells though, that is a problem

[viral sonnet]

@amber flicker basically that's already happening

what's also problematic is you parallel the spells, then you end up with 1-n targets. then the attack results (hit, crit, miss, dodge, etc..) have to be rolled which needs the stats of the caster/spell and of the target.

and as I said, previously dedicated jobs were doing this one after another. but the how to get target data from one job to the next?

it's involving write to an array/collection and that is so much slower than staying on the stack

[robust scaffold]

The application of a spell from the caster to its target will always be slow. No way around the required random access and data transfer. But I dont see why the calculation of the effect of the various spells on a single target can not be done in parallel

If you transfer all relevant data of the caster to the target in the initial random access and data transfer, the processing of all spells on the target can then be collapsed independently of other entities

[visual tundra]

If you swapped out "data oriented design" for "Object Oriented Programming", you'd be right back where/how we got here 😉

[sinful cipher]

Does each (job-)thread have its own piece of memory so that it can allocate some within a running job without having to lock anything? Is there perhaps a place where I can read up more about this?

[robust scaffold]

Welcome to DOTS. There is no documentation. What you are asking about is how Allocator.Temp works.

[viral sonnet]

@visual tundra ha, that's correct. only that cpu and memory speed changed and now memory is the bottleneck

[robust scaffold]

And look up JacksonDuncan. He did some digging into that.

[visual tundra]

You can use Temp allocations of NativeArrays in Jobs for this kind of thinking/approach, but also features of C# that don't create garbage (reference) objects - things like structs and local variables of structs, and Span<T> in the latest Unity, and the StackAlloc'ing of stack temporary memory, in any version of Unity.

[sinful cipher]

I see. I noticed how slow large native array allocations can be on the main-thread, so it felt almost too good to be true to just tuck it all away in some job running on another thread. Felt like there must have been a catch somewhere.

[viral sonnet]

the part with random access is the largest one, the rest is pretty minor. I could bring this into another stage but I'd need to know how to handle the data exchange between these 2 stages. I have done this previously like that and I used DynamicBuffers, then NativeStream and now I scrapped all that and stay in 1 single loop

[visual tundra]

if you turn off initialising of NativeArrays, they're allocated much, MUCH faster, as it makes the pointer only.

[viral sonnet]

staying in 1 job is by far the fastest

[robust scaffold]

One single threaded loop?

[sinful cipher]

Oh right I often forget about that one 😬 I should remind myself more often. Thanks

[viral sonnet]

eh, in 1 job which is parallel

[visual tundra]

https://docs.unity3d.com/ScriptReference/Unity.Collections.NativeArrayOptions.UninitializedMemory.html

[sinful cipher]

Also, lifetime of Allocation.Temp is the shortest, followed by Allocation.TempJob, right?

[viral sonnet]

also, 1 spell was previously an entity with icomps and buffers. that was quite terrible. 😦 instantiation, deletion was massive. processing wasn't much better either. now spells are basically just simple create data and a pointer to a blob

[visual tundra]

I think so. One of these is for up to 4 frames, one is just inside the method/Job. I always forget which is which.

[sinful cipher]

same, haha

They ought to rename these

[robust scaffold]

Temp is cleared at the end of every frame, regardless of if you disposed it or not (calling dispose() on a temp allocation does nothing). TempJob has some checks for frame count and memory stack is more compressed. Persistent is located on a general very large stack.

I think TempJob and Temp share memory stack...

[visual tundra]

can you print this into my brain?

[sinful cipher]

About Temp being cleared automatically - Interesting! I did not know this.

[robust scaffold]

Temp being "managed" automatically is both a blessing and a curse.

In fact, from testing by people in this discord, a temp allocation gets invalidated after a job "chunk" is completed.

[visual tundra]

Think it's a stackalloc, wrapped. Which means that as soon as it's out of scope, it's gone.

[sinful cipher]

It kind of makes it sound its tied to the scope (the calling method) in which it is allocated

@visual tundra yeah

[visual tundra]

I think you said it better than me!

[robust scaffold]

Yea. But unity also puts some specialized reactions connecting with Burst and the JobSystem so it's not a pure stackalloc

Its a stackalloc to a specific preallocated memory chunk to enable better memory linearization and vectorization.

[visual tundra]

I suspect TempJob is also a wrapper of Stackalloc, that uses some kind of voodoo to keep it in scope for a few frames.

[robust scaffold]

Yea. If the job lasts longer than 1 frame, temp remains in scope and usable

[sinful cipher]

Probably a bit more going on with TempJob as it will be accessed across multiple threads

[visual tundra]

Pinned? Fixed?

[robust scaffold]

TempJob is basically Persistant but the "distance" between the end of one allocation and the start of another is "closer" than Persistant.

[sinful cipher]

Not necessarily? I mean just standard locking by thread to prevent concurrent read/writes

[robust scaffold]

JacksonDuncan explained it a lot better

[sinful cipher]

I'll try to find the relevant article

[robust scaffold]

It allows for a lot faster and reliable access (less cache misses) than a persistent allocation

[robust scaffold]

That's just NativeCollection safety checks. The allocator doesnt care about thread safety.

[sinful cipher]

Damn everytime I return to https://www.jacksondunstan.com/ I realize it has more information than I remember

[robust scaffold]

You can do a lot of crazy shit with tempjob allocated unsafeLists.

[sinful cipher]

yeah you got a point

[visual tundra]

where do you get unsafeLists from?

[sinful cipher]

lmao look at all it

[robust scaffold]

The Unsafe version of Native containers

NativeList = UnsafeList and so in

[visual tundra]

that's in the package, right? I don't have that installed and not getting an autocomplete for nativeList

[robust scaffold]

JacksonDuncan unfortunately stopped doing things with Burst and DOTS. Real sad. Information still accurate because DOTS hasn't been updated in a while. Burst info a bit outdated.

[robust scaffold]

Yep. Its in the Collections package. Bundled with DOTS.

If you have NativeArray, you have NativeList

If you dont have NativeArray, you dont have DOTS

[viral sonnet]

is an IJobParallelFor itself able to be vectorized or just the innerloop?

[visual tundra]

NativeArray is included without Jobs, Burst and DOTS/ECS in 2020.LTS

[robust scaffold]

Yes.

[haughty rampart]

native array is in the Collections package

[robust scaffold]

Why would native array be shipped with Core? Without Burst or Jobs, it cant be used. I'm 99% sure the allocators aren't shipped with Core and without them, you can not create a NativeArray

[sinful cipher]

Mesh has SetVertexBufferData which accepts a NativeArray

[visual tundra]

I must be pulling off a miracle, then. I don't use Jobs and Burst most of the time, and don't have it installed, but I do use NativeArrays, so that when/if I want to use Jobs, I don't have to switch from builtin arrays.

[robust scaffold]

Burst documentation has an example where two arrays are added together. So I'm pretty sure ParallelFor itself is vectorized.

[robust scaffold]

My computers on campus and I'm quite far from checking my IDE for the various packages. Either way, the Unsafe versions are found inside the Collections package.

[sinful cipher]

So I guess they didnt want to have separate utility/extension methods for Mesh in yet another package

[visual tundra]

this, and for Unity Mathematics, and for the other buffers, I think the reasoning was "let's put a lot of the collections and low level stuff straight into the core, cause it's generally useful." or something to that effect.

[robust scaffold]

Note. I've done testing with IJobEntityBatch and its Execute() is NOT vectorized. For loops must be located in a separate static bursted function to become vectorized.

[robust scaffold]

Wait fuck, no. I remember now. I did some testing on IJobParallelFor a while ago and cross Execute() is not vectorized because it can be split into multiple threads. If you want vectorization, you need IJobParallelForBatch

And the For() loop within the Execute() is vectorized

[visual tundra]

Just checked, for sanity's sake, and so I don't have to suspect more nefarious activity from Package Manager... yes, NativeArray, and much more in the Core module: https://docs.unity3d.com/ScriptReference/Unity.Collections.NativeArray_1.html, in Unity.Collections, which is (of course) different from the Package Collections. Unity's naming wins, again.

[robust scaffold]

What the fuck Unity

[viral sonnet]

thought so, I'll try IJobParallelForBatch

what would you say a good batch count is? at least 4?

[haughty rampart]

whatever makes most sense for how many entities you have

[viral sonnet]

hm, something is holding this back from being vectorized

not really obvious to me what that could be

hm, I hardcoded the batch count now in addition to the check. what was 10ms is now 0.28ms ... hm, not sure if this is real

yeah, something broke due to a change. not getting any vectorization or better timings :?

maybe I need to unroll the loop myself?

[robust scaffold]

Have you checked the Burst inspector? Purple is good. Purple means vectorized operations.

Although with that code, it looks like the for loop is being deleted. You're not doing anything within the loop (everything else commented out).

Give me a few minutes. I'm headed back to my laptop and I'll post something that's vectorized

[viral sonnet]

yep, i commened out the CalculaeSpell part because the Burst inspector output was huuuge 😄

i'll check for purple stuff

[robust scaffold]

Burst is really picky on what us and can be vectorized.

[viral sonnet]

can something that is not an arithmetic even be vectorized? you see i just make a method call

nothing purple with that code ... hm

[robust scaffold]

When ya think about it, everything is ultimately math in the end...

[viral sonnet]

maybe it gets culled when i uncomment

that's also not it, there was some directive i could give burst i read in the forum. i'll try that next

Unity.Burst.CompilerServices.Loop.ExpectVectorized(); - The type or namespace name 'Loop' does not exist in the namespace 'Unity.Burst.CompilerServices' (are you missing an assembly reference?) - damn is this gone?

it's still in the 1.6.1 guide

ah, I need to set the UNITY_BURST_EXPERIMENTAL_LOOP_INTRINSICS symbol

[robust scaffold]

Also that vectorization check doesnt work for all vectorized operations. Some it will throw an error stating that it detects non-vectorized but when you go into the inspector, it'll show up vectorized.

Basically, you must move the vectorized for loop outside the main Execute() function for Burst to vectorize it. Really annoying

[viral sonnet]

thanks for the code. yeah, i get the error message that it's not able to vectorize

[robust scaffold]

Same exact code, located on a separate function. One is vectorized, the other not

[viral sonnet]

i'll keep that in mind, maybe i get it to work.

[robust scaffold]

ExpectVectorized() only really helps in really simple, maybe 1 or 2 line for loops

Good for a first step to understand what can be vectorized, what is vectorized, and how it looks like in the inspector

For example, my VectorizedAddition() function will pass the ExpectVectorized check

[robust scaffold]

But something like that will not pass, yet it is vectorized according to the burst inspector (3+ hours digging through and profiling these operations)

And yea, branches, even inlined conditionals, absolutely massacre performance

[viral sonnet]

does it have to be static?

[robust scaffold]

The function itself does not need to be labeled "static" but it can not access variables outside those defined in the parameter list. So yes. It needs to be static.

You must pass all relevant information into the vectorized function using the function parameters.

[viral sonnet]

any reason you pass the pointer and not the ref NativeArray?

[sinful cipher]

Ok before I spend too much time finishing something that has no chance of working...

[SEE SCREENSHOT FOR CONTEXT]

Am I supposed/allowed to add a chunk component to NEW entities on a per-entity basis like this?

[robust scaffold]

Burst's auto-vectorization works entirely on pointers. It doesnt seem to understand that ref NativeArray<float> is identical to float*. The pointer will get vectorized. The ref Nativearray will not.

[viral sonnet]

ah, good to know! damn, I guess I have no luck then with reading the NativeStream

[robust scaffold]

No. Chunk components are set onto "chunk" types. Not Entities.

And that is a really odd way to build an archetype

[robust scaffold]

ECBs can not set chunk components due to the fact that ExclusiveEntityTransactions unable to access chunk components (no clue why).

[sinful cipher]

So I must first do a query on entities that already exist? Also, arent entities shifted around between chunks as entities are added or deleted?

[sinful cipher]

Haha, I know. I got sick of dealing with the parentheses so now I just chain the statements like this: .AddComponent().AddComponent().AddComponent(). Then finally an implicit cast turns it into the EntityQuery.

[robust scaffold]

Yes.... kinda. I believe the intent is to use IJobEntityBatch / IJobChunk which has access to the chunk components that you then can set using the batchChunk.SetChunkComponent()

It's really annoying because I too have chunk components I need to set at creation

EntityManager itself can set chunk components but you need to first get an entity then get it's chunk then set the chunk component

[robust scaffold]

EntityManager.SetChunkComponentData(EntityManager.GetChunk(Entity), IComponentData);

[sinful cipher]

Can I expect entities to remain in the same chunk for every frame/update?

Or are they dynamically shifted around to account for 'holes' where like deleted entities used to be or something.

[robust scaffold]

If you do not do any structural changes: adding entities, changing components, changing shared component data, removing things. Entities will not move.

[sinful cipher]

Although as long as I update the chunk components to their newest state before doing any other queries on the entities, I should not have to worry about this, right?

[robust scaffold]

If you delete entities, then ya, things will change. Entities will not shift around to "fill" holes. They remain in place.

Thats why you can store an Entity as a property of a component and have it remain valid even if you're adding and removing other entities

[sinful cipher]

Man I wish Jackson Dunstan had an article on chunk components

[robust scaffold]

It doesnt seem that hard to understand. I use chunk components everywhere. But yea, adding, setting, and removing them are a real pain and only viable in struct based entity jobs.

[sinful cipher]

Maybe its because I'm trying to solve the wrong kind of problem with them

[robust scaffold]

Chunk components are like "shared" components except only shared with entities within that chunk and is readable inside a job

And dont fragment your chunks (good and bad)

god I wish there was a shared chunk component

My example of chunks, you can even see it above in my previous screenshot.

I have a group of people, all citizens of a single country. So there is a shared component with a single property that is an entity to the country they are citizens of. There's also a chunk component that is a mirror of that shared component

In the job, it determines the interest on loans from the national interest, located on the country entity

Only way I can obtain that is from the chunk component