archived-dots

[coarse turtle]

im fine with it 👀 I don't think there's a way out of using managed operations for read/writing files in c# 🤔

[white island]

i'm using a queue because as far as I know i'm only adding and removing from the end, so no point in keeping a list which is more expensive

[rotund token]

hmm from memory a* requires something like priority queue

to order by best estimated cost

the last node you add to the queue more often than not will not be the next node you want to explore

[north bay]

Hmm I personally would tend to prefer byte dynamic buffers in which the write/read data is stored
Buut that's really just preference I guess

[rotund token]

yeah so that's actually how i originally did it

just wrote to a buffer or read from a buffer

the problem was, this doesn't allow me to easily do multiple save / load requests at once

(obviously i can make more than 1 entity)

[north bay]

Uhh why not?
You can have a save or load buffer per entity, then you can represent the load/save operation as this entity

Or do your load/save operations require multiple arrays?

[rotund token]

let me just have a quick think why i removed this

because this is what i originally had

i think i had some annoyances with cleaning up the memory, taking 2 frames on one of the operations depending on the ordering of systems

but maybe with some strict ownership rules
i.e.
saving - file write system responsible for cleaning up buffer
loading - save system responsible for cleaning up buffer

i can probably avoid this

because i agree, this is a much nicer approach

do need another component to provide meta data (i.e. requested save slot) but apart from that i cant think of anything else that made me change approaches before

oh yeah there was 1 thing

the giant memcpy to the buffer

(ok not that giant)

(it's only a few MB even on 150k entities)

[north bay]

I think it could be interesting to have your save operation as one entity and then the individual 'save sections' as child entities
So that you can avoid having one giant buffer and even do things like per 'chunk' (not necessarily entities chunk) save files

[rotund token]

oh thats not the issue
everything is already merged to a single native list before this

the problem is subscenes

i support dynamically saveclose/openload on subscenes and this stores each subscene into their own collection

when saving the entire world though, a lot of subscenes might be closed and you can't save those (and i don't need to save those since I already have the collection of data)

so i'd need to merge all the subscene collections + general world collection into this single buffer

i can do this in a collection of jobs though so i don't think it's that problematic

[white island]

can i see this?

[north bay]

That doesn't sound too bad to me

Will it be possible to do partial world loads/saves with your system?

[rotund token]

yep

you can filter what you want to save

                .WithAll()
                .WithAny()
                .WithNone()
                .WithSavers() // custom savers outside of components/buffers
                .SubSceneSaver() // 1 of 2 modes, either prefab (world) or subscenes
                .Create();
            
            // Also for targeted subscene saving
            saveProcessor.SetSharedComponentFilter();```

so default setup is

            this.saveProcessor = new SaveBuilder(this).Create();```

a subscene and just regular world processor

[north bay]

Niice, I was thinking in terms of world rollbacks
Another advantage of having the data in a dynamic buffer is that it's really easy to attach meta-data to the load/save operation

[rotund token]

then to save a specific subscene
this.subSceneProcessor.SetSharedComponentFilter(new SceneTag { SceneEntity = section.SectionEntity });
just filter on it

[north bay]

So you could keep the last x world copies in memory and address them through some meta data

[rotund token]

oh yeah it can totally handle rollback

i already tested this

https://www.youtube.com/watch?v=NIE2DMxBCm4

this is just showing off sub scene save/load

but i do show just the load option

which you can clearly effectively just does a rollback

[north bay]

That's really cool!
I wrote a very hacky replay system on top of DOTS Netcode some time ago which I would love to cleanup or replace 😉

[rotund token]

i really like how this turned out

with the ability to save specific components just with the [Save] attribute

but also the ability to ignore files with [SaveIgnore]

[north bay]

What kind of compression do you use?

I had very good results with zstd

[rotund token]

im just using the lz4 that unity provides (internally)

Unity.Core.Compression.CodecService

works in burst out of bat so i didn't have to think about it

honestly file size hasn't been much of a concern for me

150k entities was ~10MB before compression (though to be fair i was only saving 4 buffers with like 2-6 elements and 8 components)

but most people won't be saving that much anyway

[north bay]

Damn I can't find my numbers anymore but I initially also used LZ4 but definitely had better results with zstd
Yea in my case file size is a concern because I'm writing out 30 snapshots per second

[rotund token]

originally i was just doing compression on final array but that takes a bit too long imo for large sets (like 8ms)
so i changed to compress per component type so i can do it multi threaded

i tested then doing a second compression and it improved size by like 20% but i dont think it's worth time so i removed it

but yeah its only getting like 50% compression atm

but the serialization doesn't require any main thread or Completes() and is fast enough I can run it in the game world on 150k entities per frame without issue

objective was to make serialization fast with deserialization being slower

though for subscenes open/load this isn't the case as the main slow part of deserialization turned out to be just creating the entities

which for subscenes you dont need to worry about

[north bay]

When are you applying the changes to the subscene entities?
Like is your save system applying it in that subscene staging group or will you have the 'initial' state entities for a tick or so?

[rotund token]

for loading?

[north bay]

Ye

[rotund token]

i run right after SceneSystemGroup in InitializationSystemGroup

and check every frame if the subscene has loaded so i can apply changes soon as it's there

to avoid 1 frame of entity jumping positions etc

which looked horrible until i got this timing working

[north bay]

Ah I found it
The 'staging' group is called ProcessAfterLoadGroup
Might be worth checking out🤷‍♂️

[rotund token]

this only runs on subscene load though

it means i'd have to split my save system into 2

which is not the worst thing

but i didn't know ProcessAfterLoadGroup existed

so that is definitely good to know

[devout prairie]

I was just looking at the videos on your channel, the voxel engine looks nice did you take that much further?

I have a friend that's made a basic voxel engine and was saying he could probably really benefit from ecs or even jobs

[rotund token]

ooo i dont think it progressed much more than that
that was a long time ago

[shut pewter]

I'm working on the effect system for my skill system. I have plenty of unique effect types (around 145), and one skill can apply multiple effects to a target (example: A heal effect + a dispel effect).

Currently I have a "dumb" setup:

• HealEffectComponent => HealEffectSystem
• XXXEffectComponent => XXXEffectSystem

I'm assuming there must be a better way of doing it, but I'm not entirely sure. In objects that'd be rather easy, each effect class has an execute method. But with ECS, I'm trying to think about adding a list of effects to apply to my entity, yet I'm not able to come up with a clean way of doing it

Interfaces maybe ? But how could I store a list of interfaces on my entity, since size isn't predictable

[rotund token]

i have written a completely new voxel engine since though that isn't tied to blocks that I will probably actually keep developing at some point as I'd like to use it

[devout prairie]

did you find doing custom meshing was faster than just basically rendering a bunch of cubes?

[rotund token]

oh much

[devout prairie]

ah really

[rotund token]

at the time in that demo though I think I actually stuck with MeshRenderer (I had a toggle to switch between hybrid renderer v1 and MeshRenderer - i still do in my new engine for debugging.)

because it was like twice as fast

i haven't tried that demo on hbr v2 though

[devout prairie]

i was thinking, the whole dots thing is rendering a bazillion cubes so i figured that, by default it'd be prime for a voxel thing

yeah maybe v2 would make the difference as i think v1 had some issues with basic stuff right

[rotund token]

im still not sure v2 is great for unique meshes

but it at least might be on part with regular renderer but i haven't tested

[devout prairie]

my thought was basically, if you have say 5 block types, just render a bunch of instances and hey presto, easy performance

maybe not in reality though, i just figured that was pretty much the whole idea with dots and hr

[devout prairie]

Sorry didn't mean to hijack!

[shut pewter]

No worries this is a public place lol

[rotund token]

this is a pretty loaded topic around here with a lot of implementations floating around and there's really no best way to implement it. but just to point out something, you could argue each 'effect job' is the same as each 'effect class'

[rotund token]

unfortunately there is still reality of 10000s of draw calls

each 32x32x32 chunk is 32768 cubes

[devout prairie]

doesn't instancing roll that into single calls though no?

[rotund token]

from memory batches can be 1024 max so that's 32 draw calls per chunk (before you add like 4x as much from shadows/post processing)

assuming single mesh

also you're now drawing 786,432 verts per chunk

[devout prairie]

well not single but muchly reduced 😛

[rotund token]

so at the very least you need to cull cubes that aren't visible

[devout prairie]

yeah that would def be my first thought, culling

an interesting thing i read about a while back was also hard edges when shading..

[rotund token]

i mean this is kind of what nanite does

but even nanite is limited to 2,000,000 instances

which would only be a 125x125x125 sized world of cubes

not a very big world

[devout prairie]

so for example if you have a cube, that's 6 sides with hard edges, on the gpu each pair of triangels sharing a hard edge is split so it has double the amount of normals along the shared edge if that makes sense.. whereas with a 'smooth' shading model where ( i think unity calls them tangents right ) are shared across multiple tris it's more efficient

so i think technically having that low poly hard edged shading is more costly in terms of gpu than smooth shaded

i guess that's something that might add up over 1000's of objects

[rotund token]

i am painfully ignorant of the gpu so i'm not going to be that helpful here

it's something i've been wanting to do a lot more work with for a while but i don't have time to generalize. i'd rather master 1 thing.

[devout prairie]

not sure if it's something that's been optimized a way in more recent drivers but it's pretty interesting

not sure if it's actually cheaper to map a single cube normal map texture over a smooth shaded cube for example, than just apply a color and make it hard shaded, if that makes any sense

[devout prairie]

yeah true, problem with being a generalist in my case, master of none 😛

[gusty comet]

I have a city builder that has procedurally generated buildings that can have variety of behaviors (in varying amounts). I can see how this can be implemented in traditional sparse storage ECS like Entt, but can't really grok the archetype version. I'm assuming this type of procedurally unique entities are undesirable with Unity ECS, right?

[devout prairie]

i would say something like that could probably still conform to ecs but i'd guess it comes down to how many of each thing you have etc.. in terms of achetypes maybe someone else could help but i'd probably think of it in terms of jobs/systems which iterate over certain groups of components.. so you could have a system that iterates all buildings with say a landing pad component and does some logic, another system that loops over all buildings with a back door, etc

[gusty comet]

Can I simply add every type of component and disable/enable them at runtime? I remember reading somewhere that is the preferred case since unity ecs doesn't really expect archetypes to change? (paraphrasing)

[devout prairie]

i think that's definitely an option.. afaik they are going to add an enable/disable flag to avoid the problem of having lots of structural changes when components/tags are added/removed often

so i do know people have recreated this manually by just having a bool in the component

so effectively you could have one building archetype, with all possible components, and it skips what it doesn't need to process

[worthy rampart]

I don't know how hybrid renderer does this but I was able to push several million cubes through the renderer using draw mesh indirect

https://media.discordapp.net/attachments/497874081329184799/855528272115924992/Desktop_2021.06.18_-_14.51.37.02.mp4

I think by default batching is 1023 instances

But if you populate a buffer and call the indirect call it can be as big as you want

[rotund token]

yeah you'd have to do it outside of hybrid renderer

thats a pretty cool demo though

are you doing the culling on a per block basis?

[white island]

Ok, I may need to return to blobbable assets. If I had a large Dictionary / Lookup table of IDs and their positions in the scene (and other scenes), currently a static dictionary, could I somehow make it into a blob asset in order for entities to be able to read and modify it?

and would Monobehaviours be able to read it as well?

this dictionary is a lot simpler than my complex structure I was trying to create when I was trying to make a dwarf fortress clone a few months ago. This is just a key (a string containing an ID), a string with a world name (basically what scene it's in), and a float3 position

well. an int2 cell position and a float3 position within that cell

both of those are blittable though

There is a Native Hashmap, but I don't yet know for certain how many IDs I need to track, so allocating it would be difficult

actually am I allowed to just do this

public struct RefIDpositionInfo
    {
        public FixedString32Bytes world;
        public float3 positionInWorld;

    }
    public static class RefIDTracker
    {
        public static NativeHashMap<FixedString32Bytes, RefIDpositionInfo> refIDLUT = new NativeHashMap<FixedString32Bytes, RefIDpositionInfo>(1024, Allocator.Persistent);

        //TODO: deserialize

        public static void updateRefIDPosition(string refID, string w, float3 p)
        {
            if (refIDLUT.ContainsKey(refID)) //if in the table
            {
                refIDLUT[refID] = new RefIDpositionInfo
                {
                    world = w,
                    positionInWorld = p
                };
            }
            else //if not, add it to the table
            {
                refIDLUT.Add(refID, new RefIDpositionInfo
                {
                    world = w,
                    positionInWorld = p
                });
            }
        }
    }

will the native hash map expand if needed?

[rotund token]

yes

but only the non-parallel version

that said, i am concerned about this class

anynative container allocated in a field is instant red flags

how are you disposing it

[white island]

It’s persistent so I’m not

[white island]

Oh, really? Why?

[rotund token]

you still have to dispose persistent

[white island]

oh... when? when the game shuts off?

[rotund token]

yes

[white island]

perhaps it should be a singleton then

[rotund token]

in theory it's probably fine in a windows build, but certain platforms (consoles) not disposing will cause an app crash on close and fail trcs
also you will get constant leak errors in editor

[white island]

and then OnDestroy() it cleans itself up

[white island]

something like this perhaps?

public sealed class RefIDTracker : MonoBehaviour
    {
        private static readonly RefIDTracker instance = new RefIDTracker();
        public static NativeHashMap<FixedString32Bytes, RefIDpositionInfo> refIDLUT;

        static RefIDTracker()
        {

        }

        private RefIDTracker()
        {
            refIDLUT = new NativeHashMap<FixedString32Bytes, RefIDpositionInfo>(1024, Allocator.Persistent);
        }

        public static RefIDTracker Instance
        {
            get { return instance; }
        }

        public static void updateRefIDPosition(string refID, string w, float3 p)
        {
            if (refIDLUT.ContainsKey(refID)) //if in the table
            {
                refIDLUT[refID] = new RefIDpositionInfo
                {
                    world = w,
                    positionInWorld = p
                };
            }
            else //if not, add it to the table
            {
                refIDLUT.Add(refID, new RefIDpositionInfo
                {
                    world = w,
                    positionInWorld = p
                });
            }
        }

        private void OnDestroy()
        {
            refIDLUT.Dispose();
        }
    }

if it's a singleton, that should work?

Oh right, c# has destructors, maybe I should do that instead

[haughty rampart]

hey people. 0.50 broke some project functionality of mine.
i want to switch out meshes on entities at runtime
prior to 0.17 rendermesh.mesh = new Mesh(); worked
with 0.17 that did not work anymore and i switched to .SetSharedComponentData(entity, new RenderMesh { [...] });
with 0.50 that broke as well. apparently it does still change the mesh when looking in the inspector, but it won't render anything
(renderMesh.mesh.SetVertices(); etc...on the original rendermesh does still work, but is highly undesirable)

anyone know what might be the problem?

[haughty rampart]

no. OnDestroy() would be the correct approach

[rotund token]

are you trying to switch 1 entity or a bunch of them

because

.SetSharedComponentData(entity, new RenderMesh { [...] })
will only update one of them while
renderMesh.mesh.SetVertices()
would update all of them

so not exactly the same behaviour

prior to 0.17 rendermesh.mesh = new Mesh(); worked
no idea why this would have ever worked tbh, you'd only be changing the local instance

[haughty rampart]

it's a terrain generator, so each chunk has it's own mesh anyway. unfortunately unity even removes mesh renderers that have no mesh assigned so i have to assign default unity mesh, e.g. cube, and then replace it with a new mesh at runtime, set the generated data, and well...push that to the rendermesh

[haughty rampart]

idk, i realize that but it did work at some point so i have no idea. was in the early testing phases though and someone already pointed it out to me, probably you. all i know is that it did work at some point but it never made sense why this would've worked in the first place

[rotund token]

unfortunately unity even removes mesh renderers that have no mesh assigned
interesting did not know that

i pool my rendermeshes in my own terrain generator

but i mark them disabled

so whatever removes them i assume wouldn't affect this

[haughty rampart]

yeah if you don't assign a Mesh OR Material to a meshrenderer, all the needed renderer components will not be added to the entity

[rotund token]

oh not using conversion?

[haughty rampart]

yeah using the conversion workflow

[rotund token]

i just convert a prefab with a material and quad

the instantiate that as needed for more instances

[haughty rampart]

yeah, same pretty much

exactly

[rotund token]

anyway to your problem hmm

not sure

i dont recall if i've tested my terrain gen since 0.50 has been out

only place i would do the same thing

i can go load it up see if i have same issue

the fact it's still targeting 2021 means it probably hasn't been updated 😄

[haughty rampart]

in the inspector the mesh gets changed, i can double click the mesh and inspect it, fully working still, but nothing renders.

using subscenes btw

not tested without subscene

[rotund token]

is it marked as static or something? hmm

[haughty rampart]

no

btw, i can't wait for unity to support c# 10. that's gonna be so great for dots projects
instead of new RenderMesh { mesh = mesh, castShadows = renderMesh.castShadows, layer = renderMesh.layer, material = renderMesh.material, needMotionVectorPass = renderMesh.needMotionVectorPass, receiveShadows = renderMesh.receiveShadows, subMesh = renderMesh.subMesh }; you can simply write renderMesh with { mesh = mesh };

[white island]

ok, so how can I read refIDLUT from this inside of a foreach in a system? even with a getter function it tells me ```The managed class type Unity.Collections.NativeHashMap2<Unity.Collections.FixedString32Bytes,IOD.Systems.RefIDpositionInfo>*is not supported. Loading from a non-readonly static fieldIOD.Systems.RefIDTracker.refIDLUT` is not supported

do I... copy it outside of the foreach loop, and pass it in?

that seems like a lot of GC work

[rotund token]

what gc? it's a native container

[white island]

or rather, a lot of allocation and disposal

[rotund token]

you're not allocating anything either

its just a struct on the stack

[rotund token]

ok updated my terrain system to 0.50 and i seem to have no issue

though i've realized i'm not doing SetSharedComponentData

        {
            var available = math.min(required, this.freeMeshQuery.CalculateEntityCount());

            // If we don't have enough meshes, create new ones
            for (var i = available; i < required; i++)
            {
                var meshEntity = this.system.EntityManager.Instantiate(this.prefab);
                this.system.EntityManager.AddComponent<Disabled>(meshEntity);

                var renderMesh = this.system.EntityManager.GetSharedComponentData<RenderMesh>(meshEntity);
                renderMesh.mesh = new Mesh();
                renderMesh.mesh.MarkDynamic();
                this.system.EntityManager.SetSharedComponentData(meshEntity, renderMesh);

                this.allMeshes.Add(meshEntity, renderMesh.mesh);
            }
        }```
I create render meshes once

then just re-use the mesh

[white island]

hmm. so if I do

Update(){
  NativeHashMap<blahblah> LUTdata = RefIDTracker.refIDLUT;
  Entities.ForEach((ref NPCWhereabouts whereabouts) =>
            {
              //do something with LUTdata
            }
}

it would... allow that?

[rotund token]

yes thats fine

[haughty rampart]

yeah persistent native collections are a bit weird to work with

[haughty rampart]

well then i have no idea what could be wrong. maybe it's just my unity project.
(each time i open the project it complains about not finding the type UIDocument and i have to change a variable name, switch to unity, let it recompile, and it works perfectly, no errors. so maybe there's more broken on the unity level and i have to create a fresh project and import everything again)

[rotund token]

(each time i open the project it complains about not finding the type UIDocument and i have to change a variable name, switch to unity, let it recompile, and it works perfectly, no errors. so maybe there's more broken on the unity level and i have to create a fresh project and import everything again)

just reimport the file that complains

that's my annoying fix

(though for me i have 2 files that complain in separate assemblies so i need to reimport both of them)

[white island]

now it tells me that this function:

public static void updateRefIDPosition(FixedString32Bytes refID, FixedString32Bytes w, float3 p)
        {
            RefIDpositionInfo newData = new RefIDpositionInfo
            {
                world = w,
                positionInWorld = p
            };

            if (refIDLUT.ContainsKey(refID)) refIDLUT[refID] = newData; //if in the table
            else refIDLUT.Add(refID, newData); //if not, add it to the table
        }

gives me this error:
The managed class type `Burst error BC1042: Unity.Collections.NativeHashMap`2<Unity.Collections.FixedString32Bytes,IOD.Systems.RefIDpositionInfo>*` is not supported. Loading from a non-readonly static field `IOD.Systems.RefIDTracker.refIDLUT` is not supported
How can it not be able to read its own field? Must I make the class unsafe?

[rotund token]

you can't use non static readonly fields in a job

[white island]

Crap... how do I work around this?

I can't mark it as readonly... I need to change it in other places

maybe I can use a get private set?

[white island]

or what if I made it non static? It';s a singleton so it should still be fine

or am I able to lock mutability for a bit like rust

[rotund token]

you can't use non static readonly fields in a job

[white island]

oh

[white island]

but it is static

public static NativeHashMap<FixedString32Bytes, RefIDpositionInfo> refIDLUT;
It's just... not readonly

because I need to be able to change it in other functions

[rotund token]

https://docs.unity3d.com/Packages/com.unity.entities@0.50/manual/gp_common_patterns.html#encapsulation

[rotund token]

its not readonly

life advice

forget the static keyword exists in entities

[white island]

as in dont use it or don't consider that it's static

[rotund token]

as in don't use it

ask yourself
'would my code work if i had 2 identical worlds running at the same time?'

if not, then you've probably done something wrong

even if it's not a feature you need

[white island]

wait, so I just... encapsulate the RefIDLUT in a struct? and then, call the struct's function?

also does the fat arrow matter

Invalid managed type found for the field `t` of the struct `IOD.Mobs.AI.ECS.LUTDataCapsule`.: the type `IOD.Systems.RefIDTracker` is a managed type and is not supported

struct LUTDataCapsule
    {
        RefIDTracker t;
        public LUTDataCapsule(RefIDTracker track) => t = track;
        public void updateInfo(FixedString32Bytes s, FixedString32Bytes w, float3 p) => t.updateRefIDPosition(s, w, p);
    }

Dang it.

so close yet so far

but I can;t make it unsafe since it's a singleton

aaagh

Jobs are a puzzle

[haughty rampart]

no. tried that already. it's broken on a deeper level

[rotund token]

thats a shame

[white island]

So, how can I encapsulate this class? Or maybe since it’s a singleton I can tell it to refer to the instance? I’ll take another crack at it tomorrow…

[rustic rain]

Class? Only through class component.

[hot basin]

Can I autoconvert list/array of entities in a field?

When there is a singular entity field it can convert prefab into entity

But I can't make the same with multiple prefabs in array or something

[rustic rain]

in prev version there used to be helper function for that in authoring class

where you provide list of gameobjects to be converted into entities prefabs

and get list of entities back

[worthy rampart]

yes it's using a raycast based culling approach

I store the blocks in a bvh that I dump to the GPU for use in a compute shader that culls the instances

while functional it's not particularly realistic for use in an actual game

[rotund token]

that's awesome

[rustic rain]

Where can I control fixed step group?

I don't use physics package btw

I want to be able to tune amount of ticks/s, whether it runs as at all and all those options

[rotund token]

just get FixedStepSimulationGroup

and set its timestep

[rustic rain]

but where

oooh

it's a field?

[rotund token]

it's literally called

Timestep

    {
        /// <summary>
        /// Set the timestep use by this group, in seconds. The default value is 1/60 seconds.
        /// This value will be clamped to the range [0.0001f ... 10.0f].
        /// </summary>
        public float Timestep```

[rustic rain]

yeah, I see

I didn't realise it's all wired through fields of system group

[rotund token]

under the hood it's controlled by RateManager

which you can setup on any system group

[rustic rain]

oh god, I feel so ashamed, but after not toching Unity for 2 months, I totally forgot how to do Quaternion lerp in dots

hmm, VS doesn't seem to be able to find this method

is that some kind of extension?

[calm edge]

are you using unity.mathematics?

[rustic rain]

I do

can't even find slerp

which I do remember having

[calm edge]

ah, it's math.nlerp

[rotund token]

math.slerp()?

(or nlerp as redwyre said)

[rustic rain]

for quaternions?

[rotund token]

yeah

[rustic rain]

hm

[rotund token]

        /// <param name="q1">The first quaternion.</param>
        /// <param name="q2">The second quaternion.</param>
        /// <param name="t">The interpolation parameter.</param>
        /// <returns>The spherical linear interpolation of two quaternions.</returns>
        [MethodImpl(MethodImplOptions.AggressiveInlining)]
        public static quaternion slerp(quaternion q1, quaternion q2, float t)```

[rotund token]

where is your s?

[rustic rain]

there's only slerp?

[rotund token]

there is slerp

and nlerp

[rustic rain]

wait

nlerp is same as normal lerp?

[rotund token]

normalized lerp

        /// <remarks>
        /// Prefer to use this over slerp() when you know the distance between q1 and q2 is small. This can be much
        /// higher performance due to avoiding trigonometric function evaluations that occur in slerp().
        /// </remarks>
        /// <param name="q1">The first quaternion.</param>
        /// <param name="q2">The second quaternion.</param>
        /// <param name="t">The interpolation parameter.</param>
        /// <returns>The normalized linear interpolation of two quaternions.</returns>
        [MethodImpl(MethodImplOptions.AggressiveInlining)]
        public static quaternion nlerp(quaternion q1, quaternion q2, float t)```

[rustic rain]

yep, pog

[rustic rain]

smth weird. System is super simple

            Entities.ForEach((ref Rotation rotation, in Translation translation, in RotationTarget target, in Maneuverability maneuverability) =>
            {
                float3 direction = (target.Value - translation.Value).Normalized();
                direction.z = 0;
                quaternion targetRot = quaternion.LookRotation(new float3(0, 0, -1f), direction);

                var newRot = math.nlerp(rotation.Value, targetRot, maneuverability.Value);

                rotation.Value = newRot;

            }).Schedule();

but entity just doesn't exist in world space

[rotund token]

Is look rotation doing what you expect?

Because you're not passing in a normalised up vector to start with

[rustic rain]

yeah

without lerp

I get exactly what I want

[rustic rain]

I do

Normalized() is my utility extension

[rustic rain]

I guess I should just move my calculations into Euler

since I only have 2D space

with 1 axis to rotate

[devout prairie]

thanks for sharing, that looks great..

tbh i kindof assumed dots/hr handled this kind of stuff out of the box, i figured it was almost the hello world of dots, spewing out a million cubes..

great job on the culling etc, seems really smooth

[rustic rain]

is there built in option for dots's quaternion to euler?

[rustic rain]

Hmmm

weird thing

when I pause fixedStep group

and unpause it

it makes all those steps through paused time

I guess, it's not really gonna work out for me

[misty wedge]

What would a long running ForEach job look like (e.g. pathfinding)? Do I just schedule the job on a custom JobHandle and save it, then use it the following frame?

Would it even work? Or would a sync point invalidate the long running component type handles?

[karmic basin]

Blob assets are immutable, you wouldn't be able to modify it

[white island]

Yeah I remember that now

can you link Entities with events? Or at least, a system

[dense crypt]

Systems will work just fine with normal C# events. You can use World.Default.GetOrCreateSystem<T> to get a reference. As for something akin to event driven behaviour, check out this thread: https://forum.unity.com/threads/comparing-different-approaches-for-events-in-dots.1267775/

[rotund token]

But then you set z=0 so if z had value it's no longer normalised was what I was getting at

[rustic rain]

true

I did notice it couple hours ago xD

that was the issue

[rotund token]

Dam sleep =(

[white island]

Wait. If I’m trying to call a method in a foreach that adds to a dictionary elsewhere. It doesn’t like it, but what I could do instead is: make a temporary list of stuff I need to change, and then after the foreach I could take the temporary list and add that to the managed one

I don’t have to do any weird encapsulation workarounds

[white island]

Ok, thank you. I’ll look at this later. Using events on entities would make my life much easier

[white island]

ha! I did it! all the errors have gone away!

[calm edge]

Are there known issues with entities 0.17 in unity 2021.3? everything seems to be working...

[rotund token]

some of the debugging tools are a bit busted (it's been a while, but from memory can't change pages when inspecting buffers)

but the actual runtime code is pretty stable

[wraith urchin]

Guys some one can help me with some dots compatible shader issues 😦 ?
I have this error on the editor . Or maybe , need i to post this error on the shaders channel ?
but the shader is either not compatible with Hybrid Renderer V2, is missing the DOTS_INSTANCING_ON variant, or there is a problem with the DOTS_INSTANCING_ON variant.
Also i read it the documentation https://docs.unity3d.com/Manual/SRPBatcher.html , and make all the things that are in there to make my custom shader srp compatible 😦

[rotund token]

its a pain to manually make your shader hybrid renderer friendly

they recommend to just use shadergraph which does it for you

if you really need to do it yourself, there is likely someone around here who knows how

but in general, go look at a shader from shadergraph and the stuff it sets up with you need to mimic

[wraith urchin]

i will try to make the shader on shader graph

:3

[chilly crow]

are there any performance/design downsides of having for e.g. a single attribute buffer component vs multiple components (assuming there would be like 50+ different attributes)
e.g.

[InternalBufferCapacity(50)]
struct Attribute : IBufferElementData
{
  public float BaseValue;
  public float CurrentValue;
}

vs

struct Health : IComponentData
{
  public float BaseValue;
  public float CurrentValue;
}
struct Mana...

The reasoning I want the single Attribute component is that it makes it much easier to author new attributes in the editor without needing to resort to code-gen.
And instead of having multiple action components like AddHealth, AddMana, etc. it could just be AddAttribute which contains an index into the buffer.

[rotund token]

there are potential performance limitations

like you can't filter for a specific attribute

however i like the 1 attribute buffer pattern for a lot of things

that said
[InternalBufferCapacity(50)]
i'd probably just store it out of chunks

[chilly crow]

store it out of chunks?

[rotund token]

[InternalBufferCapacity(0)]

putting a large buffer in a chunk like that really limits number of entities you can store per chunk

[chilly crow]

the way I see it is that everything with the Attribute component would have to contain all 50 attributes anyway even if they don't use them all

[rotund token]

yeah thats fine

im considering a similar implementation

having attributes indexed to a key so i can look them up directly in the buffer

and then giving everything in the game all attributes

though i'll probably use ~5 buffers and have ~300+ attributes

so we'll see how bad this ends up being on memory

[chilly crow]

right that makes sense

[rotund token]

go check your archetypes window

[InternalBufferCapacity(0)] vs [InternalBufferCapacity(50)]

i'd imagine in general you aren't actually iterating the entire buffer and often it'll be random accessed anyway

so putting it in a chunk isn't giving that much memory benefits anyway

likely to benefit more from other components on the entity being close

apart from that, to answer original question i think this is a fine approach

though i'm not sure how explored it has been done yet

it's definitely something i'm intending to investigate further

[chilly crow]

thank you for your wisdom

it makes it really clean to create attributes with ScriptableObjects, and have the SO store the index into the buffer so you can reference the SO to grab the magic index during authoring

[rustic rain]

Is tehre any dots alternative of debug line gizmos?

So I can draw some simple line of vector for debugging purpose?

[worthy rampart]

The easiest thing by far is to just write the hlsl in a file and expose it via custom function node

You could still write the whole shader in hlsl

And not have to deal with a graph

And just use some input nodes to feed as parameters to the function

[rotund token]

you can just use Debug.Line

terribly slow though

ALINE (or your own library) is generally recommended

[hot basin]

Yep even my gpu animation instancing shader is based on shader graph. I just plugged custom function from hlsl file. Doing whole by hand is quite troublesome and you don't know when something break when updating.

[hot basin]

That's the best choice 😃

[rotund token]

<taking notes> one day I'll look at this graphic stuff

[worthy rampart]

Honestly hlsl is pretty much C

I think if you are a programmer it's easier to write the code than to think in terms of the shader graph

But shader graph gives you some nice visual debug feedback at each node if you want it

[hot basin]

The most useful thing shader graph gives you is the setup for technical stuff like here srp batcher for the hybrid renderer etc

Writing "visual" part of the shader is not that hard

[worthy rampart]

Really depends on what you are doing lol

If it's just looking up a couple of textures yea

[wraith urchin]

well guys after a night of research and not sleep xd i realize that are two solutions for the SRP not compatible CG shaders 😦

1. make the shader on shader graph and search for the equivalents implementations
2. learn hlsl over cg and make the hlsl shader
   and 3. make my best on shader graph , then copy paste the shader on a entire new unlit.shader , so can make the modifications like ZTest or ZWrite

I choose the 3 for the job xd because i dont know how is te equivalent for ZTest Greater on shader graph 😦

So i made it a portal from this tutorial https://www.youtube.com/watch?v=toQIuCtk2pI&t=628s

[misty wedge]

I'm planning my pathfinding system for my project, and just wanted to know if my approach makes sense.

I think it would be good to allow pathfinding jobs to take longer than the current frame, because I don't really need them on the frame they are requested, and I will probably have a lot of pathfinding operations running at the same time. This means I can't use any TypeHandles though, and the entity that requested the path may not even exist anymore once the pathfinding job completes.

This means I need to collect all pathfinding requests and then run my pathfinding in a normal IJobParallelFor which is scheduled on a handle separate from the system's dependency (so the job isn't autocompleted the next frame). I'm just not entirely sure how to store my result data? Could I have the result data only live inside the job, and then copy it onto a DynamicBuffer on the entity (if it still exists) once the job completes?

[misty wedge]

One thing I'm not entirely sure on how to handle is the copying back to the entity, since scheduling it isn't possible (since that would force complete the job). I was thinking of using a NativeHashMap<Entity, UnsafeList> which stores the finalized path for each entity, but I haven't worked with UnsafeList much. Can I just dereference the pointer and add it to the list? (why does NativeList.GetUnsafeList even return a *UnsafeList instead of UnsafeList?)

[calm edge]

is there a good pattern for passing data between systems, eg system1 runs job/processes, passes to system2, system2 runs job/processes, passes to system3, etc

[misty wedge]

Entities with components? 😅

[calm edge]

I have managed data, though eventually when it's all converted that would probably be the way

[misty wedge]

Components can hold managed data, even if it's not a super good idea

(also it means you can't use it inside of a scheduled job)

There's lots of different ways to do it, it really depends on the project. I've used GCHandle previously if I absolutely needed a managed object inside a scheduled job, but I would recommend against it.

[calm edge]

I mean more like System1.OnUpdate() { ... system2.AddData(thing); }

[misty wedge]

What is AddData supposed to do?

If you just want to somehow modify the instance of a system in a world, you can get the system with World.GetExistingSystem<Type>

This is done immediately however, as OnUpdate runs on the main thread

(So it cannot have any results of scheduled jobs)

[true mirage]

I cannot use NativeArray<NativeList<int3>> How can I figure it out?

I have an array with fixed size I know, but for each element, there is a list with dynamic size (unknown). Elements are added if specific conditions are satisfied

[misty wedge]

Collections cannot contain collections due to the safety system

Either use NativeArray<UnsafeList<int3>> or find some other workaround

The safety system will not inform you about race conditions or undisposed memory, so you will have to be careful

[true mirage]

Thanks, do we have another approach?

[misty wedge]

It depends on your project. Ideally you would use entities with dynamic buffers

[true mirage]

it is really a simple job

we face it always

[true mirage]

Unfortunately, I use only Jobs, IJobParallelFor and would not like to complicate it

[misty wedge]

Again, it depends on the project, but you can also use a NativeMultiHashmap

It will let you map multiple values to a single key, and will also dynamically allocate more size if needed.

[true mirage]

I think UnsafeList is perfect for my scenario

because each element (list inside the array) runs in different threads but adding elements to that list is thread safe

@misty wedge thanks

[misty wedge]

NativeMultiHashmap also has a parallel writer, it will just not auto expand as long as you are writing in parallel

[true mirage]

 [ReadOnly] public NativeArray<UnsafeList<int3>> FaceArray;

I have decorated it asReadonly attribute, it is OK because of reading elements of FaceArray because list inside is writeonly (adding elements inside it)

I cannot decorate it more elegant?

 FaceArray[k].Add(new int3(o, j, k));
                        FaceArray[k].Add(new int3(o, j + 1, k));
                        FaceArray[k].Add(new int3(i + 1, j + 1, k));
                        FaceArray[k].Add(new int3(i + 1, j, k));

k=0 I see the lines are executed but length FaceArray[0]is zero!

[coarse turtle]

The problem is that you need to access the reference of your UnsafeList, b/c UnsafeList is a pointer wrapped in a struct, properties like Length are returned as copies when you access it in your NativeArray

[true mirage]

```cs
 [BurstCompile]
    public struct ChunkJob : IJobParallelFor
    {
        private readonly int _size;
        private readonly int _sizeSquared;

        [ReadOnly] public NativeArray<Voxel> Voxels;

        public NativeArray<UnsafeList<int3>> FaceArray;

        public ChunkJob(Voxel[] voxels, int size)
        { 
            _size = size;
            _sizeSquared = _size * _size;
            Voxels = new NativeArray<Voxel>(voxels, Allocator.TempJob); //TODO
            FaceArray = new NativeArray<UnsafeList<int3>>(size, Allocator.TempJob); //TODO
            for (var i = 0; i < FaceArray.Length; i++)
            {
                FaceArray[i] = new UnsafeList<int3>(32, Allocator.TempJob);
            }
        }

I know the size of NativeArray. It is equal to size chunk size.
There are 3 loops because voxel is 3d!
I would like to parallelize the outer loop and add faces to that specified list FaceArray[k]. Face count is unknown and depends on the chunk shape
FaceArray[k] is an UnsafeList

[coarse turtle]

what you can do is write an extension for your NativeArray to read the element as a reference instead.

public static ref T ElementAt<T>(this ref NativeArray<T> collection, int i) {
  unsafe {
    return ref UnsafeUtility.ArrayElementAsRef(collection.GetUnsafePtr(), i);
  }
}

[true mirage]

A lot of work for simple task

thanks

To use this array of lists outside, I should convert it to an array or list List<int3> .
I believe if it was NativeList or NativeArray, it could be used directly

[misty wedge]

Are there any "gotchas" besides the absence of the safety system when storing an UnsafeList inside a component?

[coarse turtle]

gotta manage the memory yourself

[true mirage]

Because NativeArray<NativeList> does not work

[misty wedge]

Well yeah, but that's the same thing for NativeList

I'm asking because I've barely used them and wanted to know of any pitfalls

[coarse turtle]

well that's really the only caveat i can think of

NativeContainers are just wrappers around the same internal data structures of their UnsafeContainer versions

[misty wedge]

I assume what you wrote a bit further up is the reason why NativeList.GetUnsafeList returns an *UnsafeList and not an UnsafeList?

Since the NativeList is what is wrapping it

[coarse turtle]

Yup

[misty wedge]

I assume I can safely store a regular UnsafeList inside a component, but if I modify it I would need to pass a *UnsafeList or use ref?

[coarse turtle]

Yea

[misty wedge]

What happens if you just pass a copy? The internal current index and buffer location would still exist and be writeable, but if you then add to the original list, I assume it would overwrite what you just wrote?

[coarse turtle]

I imagine you mean UnsafeList* not *UnsafeList cause those are 2 separate things 🤔

[misty wedge]

Yeah my bad

[true mirage]

😐

[coarse turtle]

Let's say you pass in a copy, you'd still manipulate the contents of the address, but if you care about 'metadata' like Length because you want to add new content, then you have a chance of overwriting the previous content

// Assume someList is a copy of an UnsafeList<T>
someList.Add(100);
someList.Add(101);

// You expect the list store 100 & 101, which is 2 elements

// In the next frame
someList.Add(102);
// You expect the list to store 3 elements, but when you debug the Length property
// you see that it stores 1 element.
// The contents of the list would actually be [102, 101], because Add() has to read 
// `Length` in order to push a new element. Because you manipulated a copy of 
// the struct, you don't store the metadata between frames.

[misty wedge]

Isn't part of that metadata the index location in the backing array though?

Ah it reads Length, yeah, that makes sense

What happens if the addition to the copy happened to resize the list? I'm guessing the resizing automatically invalidates the old memory location?

[coarse turtle]

yea

[misty wedge]

Will it crash then?

[coarse turtle]

probably - if there's something holding onto the old pointer and attempting to access it since it might be invalidated

[misty wedge]

Yeah like a copy of the unsafe list that still has the old location and you attempt to Add

[coarse turtle]

yea

[misty wedge]

Alright, I think I get it

Thanks a bunch for answering all my questions, I really appreciate it!

[coarse turtle]

yea np

[misty wedge]

One last thing, is there any point to storing an UnsafeList* inside the component, or is UnsafeList fine?

I guess it would keep one from accidentally copying the struct

[coarse turtle]

I wanna say it depends on context, cause what I'm thinking is if you use like a ForEach lambda

struct SomeComponent : IComponentData {
  public UnsafeList List;
}
Entities.ForEach((ref SomeComponent a) => {
  a.List.Add(...); // This would be okay since you access SomeComponent by ref.
});

In a IJobEntityBatch

struct SoemJob : IJobEntityBatch {
  ...
  public void Execute(...) {
    NativeArray<SomeComponent> b = batchInChunk.GetNativeArray(SomeComponentTypeHandle);
    for (int i = 0; i < b.Length; i++) {
      b[i].List.Add(someValue); // This wouldn't work nicely, you need to access by ref
    }
  }
}

[misty wedge]

Yeah I guess you just gotta be really careful with all the unsafe collections

[coarse turtle]

personally I tried to avoid if possible, but if it worse comes to worse, then yea I'd just store a pointer 🤔

[misty wedge]

This might seem silly, but I'm just asking to make sure. To "use" the pointer then, I would use the "C++" way of doing it, e.g. myList->Add(...)

Barely used pointers in C#

[coarse turtle]

yea

or (*myList).Add(...)

I also think myList[0].Add might also work 😅

yea gets confusing, just stick with the -> operator haha

[misty wedge]

It probably will, since it would just be (*(myList+0))

Something else I was wondering today to which I couldn't find an answer. Since afaik Unity uses a fixed number of worker threads internally when scheduling jobs, do those threads need to complete before they can be reused? Or can they "halt" during execution similar to CPU scheduling with a virtual threadcount greater than the physical thread count?`

For example, what happens if Unity has 8 worker threads, and schedules 8 extremely long running jobs, would those jobs eventually yield processor resources during execution, or would the program halt until they finish?

[coarse turtle]

I think it only halts the main thread if you call complete as a sync point. You could have like a long running job by making sure that handle isn't managed in some automatic dependency chain I think. tbh I'm not really sure about if Unity's job system threads can halt and then allow you to use it for a diff job 🤔

[misty wedge]

Yeah, I was planning to have my pathfinding jobs run "outside" of sync points by not passing the system's dependency to the job, but I'm not sure how long running jobs behave if you schedule a lot of them at once

I guess it would be easy enough to test, just schedule JobsUtility.JobWorkerCount / JobsUtility.JobWorkerMaximumCount amount of infinitely running jobs and see if it freezes completely or just gets slower

[true mirage]

I keep data separately for each chunk. Inside each chunk, I have used IJobParallelFor to get faces. It is OK.
Now, I would like to parallelize chunks as well.
I cannot use IJobParallelFor inside jobs, right?
I have tested this approach. Add all jobhandles for all chunks in one NativeArray and call CompleteAll method. I do not notice significant boost
but CompleteAll does not utilize multi threading?

[misty wedge]

CompleteAll will simply block the main thread until all jobhandles are completed

[rotund token]

how many path finding requests are you making per/frame that you need to run them long?

[short hinge]

What is DOTS used for?

[upper tiger]

How do I get a box collider from an entity in DOTS

var collider = MainLoader.entityManager.GetComponentObject<Unity.Physics.BoxCollider>(icon.buildingEntity);

This throws an unknown type error

[dense crypt]

The short answer is that it makes multithreading and writing more efficient game code easier. Although since it's still early prototype it requires a lot of experience with coding and Unity to get in to.

[rotund token]

boxcollider is not a component

PhysicsCollider is the component

it has a field called Value
public BlobAssetReference<Collider> Value;

that holds the collider info in blobasset memory

[upper tiger]

ahhh thats where the blob asset reference is!

thanks

[rotund token]

to get the actual box collider you need to use unsafe code

[upper tiger]

can you point me in the direction of how to do that

[rotund token]

actually to avoid unsafe code you could try this

ref Collider collider = ref physicsCollider.Value.Value;

if (collider.Type == ColliderType.Box)
{
    BoxCollider boxCollider = UnsafeUtility.As<Collider, BoxCollider>(ref collider);
}```

make sure you return it by ref from the blob otherwise you will only copy the header to the local value and lose all the box info then read memory out of range

[upper tiger]

Thanks!

                    var physicsCollider = MainLoader.entityManager.GetComponentData<PhysicsCollider>(icon.buildingEntity);
                    ref Unity.Physics.Collider collider = ref physicsCollider.Value.Value;

                    float yOffset = 0;
                    if (collider.Type == ColliderType.Box)
                    {
                        Unity.Physics.BoxCollider boxCollider = UnsafeUtility.As<Unity.Physics.Collider, Unity.Physics.BoxCollider>(ref collider);
                        yOffset = boxCollider.Size.y;
                    }

This worked for me

[upper tiger]

Another question if you dont mind. I am trying to do a ray cast click to select an object. This normally works fine but if a UI RawImage element is on top, the raycast doesn't collide with the object underneath. I have tried to change the CollisionFilter to only collide with the desired layer and put those objects as belonging to that layer (with the Belongs To on the Physics Shape script) but this doesnt work. I even made the objects explicitly "collides with" set to a layer that i put the RayInput "BelongsTo" on and it doesnt work. what gives?

                RaycastInput RayInput = new RaycastInput
                {
                    Start = unityRay.origin,
                    End = unityRay.origin + unityRay.direction * k_MaxDistance,
                    Filter = new CollisionFilter
                    {
                        BelongsTo = 1u << 2,
                        CollidesWith = 1u << 1,
                        GroupIndex = 0
                    },
                };

I even put a physics shape on the UI element prefabs and made them belong to nothing, collide with nothing and no collisions at all and it doesnt work either!

[rotund token]

well unity physics raycast wont collide with UI

are you sure this ray is firing

generally UI that triggers a raycast will try top stop sending the input event any further

if you don't want your UI to raycast the rawimage, just turn off raycast target on the rawimage?

[upper tiger]

Nevermind! I was stopping the raycast from even occuring further up in the code if the pointer was over a gameobject. silly me

it works fine now

[remote crater]

I have a question... How do I send a commandbuffer type thing into this: https://pastebin.com/QanHcnwj

I'm still trying to upgrade to .5. I'm almost fully done.

I'm trying to get an EntityCommandBuffer type thing into a ICollisionEventsJob

Thanks for getting me this far. This is the last step. In .17 I could use a commandBuffer when I got the colliding objects so I could process them and change the state of the simulation. In .5, I think you can still do this, but it isn't obvious.

[rotund token]

i don't believe anything has changed between 0.17 and 0.50 regarding command buffers

(except the query overload for removecomponent)

[remote crater]

I'm using different collision code.

My old .17 was Jobs or something

Dependency = new CollisionEventImpulseJob
{
ed = GetComponentDataFromEntity<EntityData>(true),
td= GetComponentDataFromEntity<Translation>(true),

}.Schedule(_stepPhysicsWorldSystem.Simulation, Dependency); _endSimEntityCommandBufferSystem.AddJobHandleForProducer(Dependency);

Thats my new code. I pass in EntityData and Translation... But I am not sure how to pass in an EntityCommandBuffer or something

I can detect collisions, but really not sure how to do anything with em since I have no way of buffering EntityCommands.

I see there is an _endSimEntityCommandBufferSystem thing there, but I'm not sure if it is passed into the CollisionEventImpulseJob : ICollisionEventsJob

Its a very simple question for someone who knows... I'm just stumbling over it.

Thanks everyone for getting me this far. I have complete confidence someone will help me finally get a stable compile sometime. I'll just code around it in other parts of my projects. One key thing for game development is to never let a brick wall discourage you. Keep working at other parts.

[rotund token]

so I randomly stumbled upon a fix for this

make a build (I only tested via build configurations)

as soon as I made a build, anytime I open unity in my project I no longer have the issue with compile errors

i tested this on 2 other projects that also had the issue and it fixed both of them as well

[misty wedge]

I'm just thinking about how to go about it. You think it would be better to just finish them each frame?

[rotund token]

depends entirely how many you are doing

at work we finish them in the frame

doesn't take that long

even old ps4/xboxone handles it fine (just)

pc smashes it though

[misty wedge]

Maybe I'll go that route first and if it becomes an issue run it outside of the update loop

[rotund token]

one tip is though make sure you split your workers evenly not by chunk

if you have like 50 actors doing pathfinding

and they're all in the same chunk

and go on the same worker

but all your other workers are empty

bad things happen

[misty wedge]

Yeah I was already thinking that may be an issue, thanks!

[rotund token]

the new ScheduleGranularity.Entity can help you with this

[misty wedge]

Is this on 0.50?

[rotund token]

yes
.ScheduleParallel(this.QueryWrite, ScheduleGranularity.Entity, default, dependency);

/// <summary>
/// Describes how the entities that match an `EntityQuery` should be distributed when a job is scheduled to
/// run on multiple worker threads using `ScheduleParallel()`. In most cases, <see cref="ScheduleGranularity.Chunk"/>
/// should be used.
/// </summary>
public enum ScheduleGranularity
{
    /// <summary>
    /// Entities are distributed to worker threads at the granularity of entire chunks. This is generally the
    /// safest and highest-performance approach, and is the default mode unless otherwise specified. The
    /// entities within the chunk can be processed in a a cache-friendly manner, and job queue contention is
    /// minimized.
    /// </summary>
    Chunk = 0,
    /// <summary>
    /// Entities are distributed to worker threads individually. This increases scheduling overhead and
    /// eliminates the cache-friendly benefits of chunk-level processing. However, it can lead to better
    /// load-balancing in cases where the number of entities being processed is relatively low, and the cost of
    /// processing each entity is high, as it allows the entities within a chunk to be distributed evenly across
    /// available worker threads.
    /// </summary>
    Entity = 1,
}

this specific case is exactly why this option exists

[misty wedge]

Ah cool

Still on 0.17 but that is an easy change once I switch

Something else I'm not 100% sure on how to do, is specific to the project I'm making. I basically have a very very large world, and the server will spawn entities for stuff like trees whenever an NPC or player is near. Now for pathfinding I need to know where those trees are so I can path around them. The question is how to store those things.

My current approach is just using a NativeMultiHashMap<int2, Entity> which stores all entities per chunk, but I have no idea how multi hashmaps behave once they get very full. The other option would be to use the CollisionWorld to overlap rectangle and find all entities on a chunk that way, but it might be too slow...

[rotund token]

generally you'd write this info to your grid

i'm not exactly sure what/how your hashmap storage works with your pathfinding

[misty wedge]

The grid is too large to store it in memory

[rotund token]

sounds like it's time to upgrade from a tilebased grid!

(how big is your grid?)

[misty wedge]

You mean some other spatial partitioning?

Something like 1024*512*16*16

[rotund token]

navmesh etc

[misty wedge]

It's procedural so I can't precompute it

[rotund token]

oh thats much smaller than i was expecting

honestly i'd just store that in memory

thats only 128MB if each tile is 1 byte

if you only need a bool per tile (walkable, not walkable) and bit packed them that'd only be 16MB

unless you're targeting last gen consoles, xboxone, ps4 etc

[misty wedge]

I might need to stick to 1 byte per tile since I would like varying costs in the future, but 128mb still seems fine

[rotund token]

i don't think any other platform is going to have memory issues on that

[misty wedge]

Switch would probably

[rotund token]

yeah i count that as last gen

[misty wedge]

How much RAM do ps4 and xbone have? iirc it wasn't much since I think it was unified with VRAM?

[rotund token]

i just don't develop on it so forget to write it

ps4 you usually get around ~4.5GB
xbox usually around ~4.9GB

xbox is better as it dynamically shares its memory between gpu/cpu

ps4 you have to specify memory split so there's always waste

[misty wedge]

The game is entirely 2D and makes heavy reuse of sprite atlases, so I think I'm pretty good on memory budget. It's all simulation data

[rotund token]

honestly if you're not releasing this year i wouldn't care about last gen consoles

game sales on them are dropping fast

the effort to develop is not worth it

(switch is a different story i guess in this case)

My current approach is just using a NativeMultiHashMap<int2, Entity> which stores all entities per chunk, but I have no idea how multi hashmaps behave once they get very full.
anyway to your original point

they're fine if that's the direction you want to go

i regularly use/test maps with 200k elements

[misty wedge]

It just felt like a very lazy approach

But you are right I could just fit it in a grid

[rotund token]

do whatever is easiest and optimize it later if you need

[misty wedge]

Thanks a bunch for all the answers! Hard to find stuff for ECS 😦

Hope it will change once 1.0 comes out

Also just out of interest, do you think the collision world approach is too slow? iirc I remember reading somewhere that the spatial partitioning scheme used was very fast

[rotund token]

Also just out of interest, do you think the collision world approach is too slow?
I'm not particularly sure what you're talking about here. the bvh in the physics world is pretty fast for all that it enables. it's definitely not the fastest partitioning if you are doing something very specific but for general purpose it does well.

[misty wedge]

Basically during pathfinding whenever a chunk is visited that wasn't visited during that search, use the collision world to find all entities on that chunk

Instead of using my own grid / native multi hashmap

[rotund token]

oh if you just want to do a AABB over a giant area that's not that big a deal tbh

[misty wedge]

Yeah pretty much

[rotund token]

i think you could get away with a few 1000 of them

[misty wedge]

I already use it for some stuff, e.g. finding all relevant entities to sync for a specific client

[misty wedge]

That's more than I expected, but I'm thinking it probably won't scale for every operation that needs to check "is there a thing here?"

So I guess I'll stick with the hashmap and switch to a grid if I need to trade memory for speed 👍

[rotund token]

really just won't know till you try it

always hard to theorize how something will scale

[misty wedge]

Definitely

[rotund token]

anyway i got to go source some dinner, back in a little

[misty wedge]

But the granularity thing is really cool. If you were to schedule very "fine grained", do you think something like false sharing could become an issue? or is it outweighed by the multi threading? since like you mentioned single dangling threads with heavy workloads could become an issue since there is now work stealing or something like that.

[misty wedge]

Enjoy 🙂

[misty wedge]

This would force me to move an entity to another chunk if I want it to request a pathfinding operation correct? Since I remember hearing somewhere that moving entities is comparable expensive and it is better to keep the archetype the same and query a boolean to check if a certain operation should be done. That wouldn't work with the granularity though, since all NPCs would be in the archetype, not just those that want to find a path

[rotund token]

Well not in 0.50 you can just use the overload and unity still split it for you

In 0.17 you should use ijobfor and just pass the entity array from the matching query

[misty wedge]

Yeah but if I have something like:

public struct PathfindingComponent : IComponentData
{
  public int2 Start;
  public int2 End;
  public bool RecalculatePath;
}

The query cannot check if recalculate path is true

So either I need to run it over all entities in the archetype or run a previous job to find all matching entities

[rotund token]

Yep

[misty wedge]

I feel like enable bits would still really help with something like this

[rotund token]

Alternatively you could just make a tiny event

With the entity reference in it

[misty wedge]

Isn't that also not a good idea at large scale though?

[rotund token]

Either an Entity event with a single component entity field or something like an event system

What I'd do depends on my entity count

If you were only in dozens to hundreds per frame entity event is fine

I think you'll hit pathfinding limits before you hit event limits

[misty wedge]

That's something I'm still struggling with when it comes to ECS. Say I want to deal damage. My preferred way of doing it would be to create entities that have the target entity, damage type, damage amount etc. But the "better" way always seems to be either a simple component with a damage value that is 0 unless damage is being dealt, or a dynamic buffer with damage instances. Is that still valid even in 0.50?

(at large scale anyways)

[rotund token]

Damage has been a heavy topic of discussion recently

[misty wedge]

Does the dependency management system take IJobFor into consideration? So if I pass a ComponentDataFromEntity with write permissions to it, will it schedule it properly and alert me of any race conditions?

[rotund token]

Yes

all dependency management is handled by the system, not the jobs

[misty wedge]

Thanks, I wasn't sure.

[rotund token]

calling
this.GetComponentDataFromEntity<>()
on the system

adds the dependency to the system

[misty wedge]

Ah yeah, that makes sense

[rotund token]

only mistake you can make is calling EntityManager.GetComponentDataFromEntity<>()

which will break safety as the system won't get the dependency

(there are some very obscure reasons you might want to do this though)

[misty wedge]

Yeah I rarely if ever use EntityManager unless it's Run or WithStructuralChange stuff

[viral sonnet]

any new tricks for particles in entities? are we still supposed to use gameobjects for it?

[haughty rampart]

it should automatically create a companion-link entity iirc.

[viral sonnet]

ah right, well that can turn out helpful. although I'm still not sure how to deal with instantiating. Previously I just used pools of particle systems so I'm still not convinced which benefit particle systems in Entities could give when I'm stuck with unbursted code anyway. I'm inclined to just leave this part in MB world and pull the necessary data from the ECS world.

[rotund token]

i'm finding this very satisfying

think i'm about done with this save library, wonder what i should do with it now

maybe actually integrate it in my project to prove it works outside of demo scenes

[misty wedge]

How can I schedule a IJobParallelForDefer with a NativeArray?

[rotund token]

IJobParallelForDefer is for lists in defer mode

what do you mean by with a native array

[misty wedge]

Specifically from EntityQuery.ToComponentDataArrayAsync

[rotund token]

you dont need IJobParallelForDefer

just IJobFor

[misty wedge]

IJobFor.ScheduleParallel only takes an integer for scheduling though?

[rotund token]

yeah?

thats your array length

[misty wedge]

But won't the length only be valid once the async job completes?

[rotund token]

no

native arrays cant change in size

the array you get back from that will be the right length

[misty wedge]

That... makes sense

It's late 😦

[rotund token]

it might just have garbage data

[misty wedge]

[misty wedge]

Physicalized cubes are always satisfying

What's the use-case for the saving library?

[rotund token]

does your game need saving?

[misty wedge]

It does indeed

[rotund token]

bam use case detected

[misty wedge]

!!!

Is it custom stuff or does it use unity's serialize world stuff

[rotund token]

entire custom

i wrote a piece why you should never use serializeworld not long ago

[misty wedge]

Do you have a link? That would be interesting to read

[rotund token]

well the tldr is any change you make to any component will break all your users saves

or if you update entities

or if you change a namespace

or anything

[misty wedge]

That does seem like an issue

[rotund token]

it has no capability of migration or making changes once you release your game

so don't ever intend to update your game if you use it

[misty wedge]

Just ship the game finished and without bugs duh

[rotund token]

(it's not designed for saving, it's designed for building subscenes)

https://forum.unity.com/threads/dots-0-5-and-serialization.1258140/#post-8027630

anyway that was my post about my experiences and why i'm writing this library now

this is the 3rd time i've written a save library*

[misty wedge]

Do you know if Unity is planning on adding something actually usable? Like you showed me earlier, there are quite a few games that may want to actually save the player's progress

[rotund token]

first 2 at work, this one for my personal use

first one was using serializeworld (ok so i didn't actually write most of this it was mostly another dev, i just used it a lot and experienced it's weaknesses)

second one is doing well and has been shipped and is out in the world

and we've done multiple data migrations via patches

[misty wedge]

In the second approach, are you using reflection or is it just manually "rigging" the loaded data to the newly created entities?

[rotund token]

no reflection at all in that approach

second approach being what i did at work from memory

[misty wedge]

These archetype containers are just serialized normally? With some kind of serialization library (JSON or whatever) not using the Unity serialization system right? (Since that would break if Unity changes something)

[rotund token]

in second case?

[misty wedge]

Yeah

[rotund token]

they're unamanged, we just write the bytes directly to a stream

[misty wedge]

Wouldn't that still break if you add something to an archetype container?

[rotund token]

yeah but you can migrate it

[misty wedge]

Just making sure I understood it correctly

[rotund token]

each container is versioned

on loading save it runs a routine after deserializing into the container to see if its latest

if not migrate to next version

repeat until it's latest

simple function that just converts 1 container to next

[misty wedge]

Cool

[rotund token]

we try to save as little as possible

[misty wedge]

So similar to standard DB migration stuff

[rotund token]

so that we have to write as few migrations as possible

less you save, less issues you have

[misty wedge]

I'm amazed the first approach even worked for you, migrating the serialized world sounds like a nightmare

[rotund token]

yeah we have this dev that loves this low level type of stuff to a detriment 😅

it worked greatish but would just keep randomly breaking

[candid epoch]

is your library on github?

[rotund token]

not atm

[rotund token]

mostly nested structs turned out to be a nightmare

and any change on any component required a migration

so we were just constantly writing them

every day of work was like, write a migration 😐

too much data saved

the other issue is, can't make changes

if design change your creatures max health from say 100 to 150 for balance

that change won't be reflected in any save file until they start a new game

instead if you just dont save max health then when they load a save, it'll populate the new value instead

[misty wedge]

Oof

Your new approach sounds really interesting

[rotund token]

so yeah one flaw with approach 2 is if you make a change to a component

[misty wedge]

I'm currently using SerializeWorld, so I'll need to change to something else once saving is actually needed.

I also ran into the issue that "too much" data was saved, especially stuff like SystemStateComponents that shouldn't exist on a load. I also used attributes to strip them in a copied world before the world is serialized.

[rotund token]

and that component exists in multiple save containers

you have to migrate multiple containers

thankfully this doesn't happen much

but still something i wanted to avoid

also there is a lot of boilerplate setting up containers per type

and its a bit against a data driven design where design cant just add something new to an archetype

without a dev coming in and adding it to the save

i.e. lets say making a building grow fruit like a plant

really happy with how this has turned out now

[misty wedge]

Yeah definitely looks promising, good stuff

[rotund token]

just need to get some production testing on it

but thats unlikely to happen anytime soon, so at least get some real world testing on it

[viral sonnet]

Have you seen the GDC 2022 Entities videos? I saw they have a new api instead of Entities.ForEach and it's just Lamba with Query(...) now

Also a Baker api instead of IConvertGameObjectToEntity

[misty wedge]

@rotund token how did you handle entity reference (de)serialization in the second version of your saving system? Just save a GUID or something instead of the entity index / version?

And keep track of Guid->Entity when deserializing?

[rotund token]

we just save the entity reference

on deserialization the first step is to create all the entities which are then mapped to their previous reference in a hashmap

then during the apply step the fields are remapped

in my current (3rd) approach this is automatic now using TypeInfo

[misty wedge]

Thanks!

[rotund token]

var prefab = GetPrefab(container.Type);
var newEntity = EntityManager.Instantiate(prefab);
remapHashMap.Add(container.Entity, newEntity);```

the basic theory (obviously a bit more advanced for the sake of performance, does the whole thing in batches)

[rotund token]

was there something other than?

DOTS authoring and debugging workflows in the Unity Editor

[viral sonnet]

yeah, that's the video, the other one linked here (https://forum.unity.com/threads/dots-talks-gdc-2022.1271147/) wasn't that interesting

[rotund token]

ah yeah watched those both a few weeks ago

[misty wedge]

If I need to store a reference to a native array inside an IComponentData, how would I do that? Can I cast UnsafeList to the NativeArray type?

Since there is no UnsafeNativeArray

[rotund token]

i would first ask, why

[misty wedge]

Just easier access to a native array that lives somewhere else

[rotund token]

again why

[misty wedge]

Would you store it in a dynamic buffer?

[rotund token]

(why not use dynamic buffer [unless your enzi and you need to simulation 4bazillion things at once])

[misty wedge]

Is storing a lot of data in dynamic buffers fine? It's a 1024*512 array

[rotund token]

is this your world grid?

[misty wedge]

(not really a lot I know, but not like 5 elements)

It's a different world grid

[rotund token]

we store our world grid in a buffer

[misty wedge]

Basically references to procedural world cells for faster lookup

It's a native array of entities

[rotund token]

it depends how many systems need it

i don't like sharing containers between systems

(i have a strict, avoid all system coupling rule)

[misty wedge]

How big is the grid?

[rotund token]

ours is only 2k x 2k but we store quite a bit of data per cell

i think its like 30 bytes per cell

[misty wedge]

Still bigger than what I have, but I could potentially have multiple worlds

[rotund token]

for the record if i was around when this grid was actually created

(and i was as experienced as i am now)

i would not use a grid 😅

i'd make us use a navmesh

[misty wedge]

Like I said it's actually not the data, just a lookup from world chunk -> world cell entity

[rotund token]

yeah anyway

[misty wedge]

They're voronoi cells. Ideally I would use a better lookup scheme

[rotund token]

if you only need it in 1 system then i'd use a native container stored in that system

if i needed it in multiple systems for some reason i'd use a buffer

[misty wedge]

I'll need it in multiple definitely

Atleast the way it's currently set up

[rotund token]

for the record, i do have a dynamichashmap i released publicly

[misty wedge]

So like dynamic buffer but a hashmap?

[rotund token]

yeah

under the hood its a dynamic buffer

it just uses the dynamic buffer as a hashmap (or multihashmap)

its pretty neat

[misty wedge]

Cool, that could come in handy, do you have a link?

[rotund token]

https://forum.unity.com/threads/dynamichashmap.1256154/

[misty wedge]

Awesome, thanks.

[rotund token]

i might just update it

i forgot the set method on the hashmap

when i first uploaded this

note it won't compile out of the box as you need to give yourself a bit of internal access from memory

i intend to release my core library (this included) at some point

[viral sonnet]

I honestly don't get using DynamicBuffers. Why store something in a subpar system that is not built for it. Either store the data in NativeContainers or in IComponentData. I don't think there's any case where you'd HAVE to use a DynamicBuffer

[misty wedge]

Easier access in jobs?

[viral sonnet]

All NativeContainers are easy to access.

[misty wedge]

How do you store the reference to the container in the component?

[viral sonnet]

The only argument that I read today, from DreamingImLatios is faster building of the data. I let that slide. But with more or less static data this benefit gets marginal

If I don't misunderstand, with a pointer?

[misty wedge]

I guess people coming to ECS just aren't very used to working with pointers in C#

[rotund token]

let's see

• unable to update data at runtime in entities 1.0
• unable to use entity journaling
• coupling systems together
• the performance is actually fine unless you are doing ridiculous work loads
• saving is a pain in the ass

[viral sonnet]

I think I do misunderstand 🙂 Well, this whole binding of very global data to entities is odd in most cases. I would never store a grid as entities for example

[misty wedge]

I store the voronoi cells as entities because it just makes iterating over them very easy, but it could obviously be done in a different way

[viral sonnet]

NHM and NMHM is great for any grid data

I mean, I get why people use them. I also did because the iteration and creation is so easy. I just hope you don't get stuck with a solution that you want to change at a later point.

And were it not for the performance overhead I'd use them too 🙂 hehe

[misty wedge]

I think it's part of the learning process, and I am at that point for a lot of things.

I still have issues when I need to map "complex" things like e.g. large multihashmaps on a per entity basis.

For example, I have an entity that represents a world, and each world has a NativeMultiHashmap<int2, Entity> that contains the chunk->entity mapping as a simple spatial mapping. I'm still not sure what the best way to access this data in a job is. I'm guessing just a pointer is easier in a lot of cases

[viral sonnet]

correct, the entity is just a placeholder to poll the data you need, right?

[misty wedge]

Yes, and other entities have components that reference what "world" they are in

[viral sonnet]

get comfortable with pointers, it'll be worth it in the long run. Also, jobs that build mappings with pointers are blazing fast

[misty wedge]

Basically like a simple reference, and the world entity should contain (but doesn't) the collection

[viral sonnet]

I do this to have faster access for LocalToWorld matrices.

[misty wedge]

Any tips? I've barely worked with pointers in C#. How do they interact with generic types? Since I can't seem to get a pointer to one

Do I just use any value and make sure to properly cast it back to the real value?

(like an IntPtr or something)

[white island]

what on earth does this error mean? The type 'Unity.Collections.NativeArray<CellNavigationSystem.Node>' cannot be used as type parameter 'T' in the generic type or method 'FixedStringMethods.IndexOf<T, T2>(ref T, in T2)'. There is no boxing conversion from 'Unity.Collections.NativeArray<CellNavigationSystem.Node>' to 'Unity.Collections.INativeList<byte>'. [Assembly-CSharp]csharp(CS0315)
From this code:

public struct Node
    {
        public FixedString64Bytes cellWorld;
        public float3 coords;
        //TODO: Whitelisted covens / keys
    }

static NativeArray<Node> nodes; //Nodes in graph
Node start;
openList.Add(nodes.IndexOf(start)); //error here

[viral sonnet]

@rotund token all correct. It's not that DynamicBuffers don't have their place if you know what you're doing and keep it simple and a small memory footprint. But right now it's a pitfall for newcomers when they think of, I need an array, oh I use DynamicBuffers. Memory layout goes to shit, job overhead increases and soon lots of ms are gone just for DBs access. And as I said, a million times now. I don't understand why DBs are slow. They should be faster than NHM in theory.

[pliant pike]

that's true, that's why I try and avoid using them unless I really have to

[viral sonnet]

UnsafeUtility has most of the useful methods. For example AddressOf<T>

[misty wedge]

Ah such just a void*

[viral sonnet]

yep, they just use void pointers. Can be cast to anything you like

[misty wedge]

Why does AddressOf get the ref value? To avoid copying?

[viral sonnet]

pointer access inside IJobEntityBatch

[viral sonnet]

correct

in fact, lots of performance can be gained if you avoid "by value" and use get the reference

CDFE is one contender for screwing up performance because you get local struct copies

[misty wedge]

Is this for faster access compared to chunk.GetNativeArray?

[misty wedge]

Yeah I'm still not 100% sure when copying can be better than using in inside a job when passing stuff to functions

[viral sonnet]

yes, because you can get a ref

[rotund token]

see you keep claiming this performance thing, but i don't see any evidence. i can write a performance test in a few minutes and every time i do nativearray and dynamic buffer perform exactly the same
in fact i just did

numbers represent length buffer/array, number of entities with buffer/array

so the
bottom test is 10,000 entities with 1000 length buffers
top is 100 entities with 100,000 length buffers

it's fraction of a millisecond difference over 10,000,000 elements

[viral sonnet]

can you share the test? I can start mine again to compare

[rotund token]

yeah of course

[pliant pike]

well that's good to know

[rotund token]

the only thing i can think of is you are measuring with safety on

this is what happens when full safety is on

[pliant pike]

but the memory fragmentation can still be a problem

[rotund token]

its 2 orders of magnitude different

[pliant pike]

like storing lots of buffers on lots of different entities will always be slowly than one single list

[rotund token]

this native array test is storing it on the component so has no safety even when safety is on

        {
            [NativeDisableUnsafePtrRestriction]
            public void* Ptr;

            public int Length;
        }```

[viral sonnet]

nah, safety is off and numbers weren't that horrible 😄

[rotund token]

it's probably this different because safety breaks the simd in this test

pick it apart

make sure i haven't done something stupid

test is just summing every number in the buffer/array and writing it to an index

[viral sonnet]

I'll start up my old performance project. I've done very specific tests for reading mostly. But also some write tests

[rotund token]

i would love to see a test where the performance is significantly different

[viral sonnet]

a test script that is still in my project, DB still lags behind NMHM

[rotund token]

because from the source code i dont really understand why this would ever be the case

the trick is you obviously have to always use AsNativeArray when reading/writing

otherwise apart from having to do a condition check every time, it'll never optimize

which a lot of users won't know by default

[misty wedge]

That's good to know 😅

[rotund token]

because it has to do this
UnsafeUtility.ReadArrayElement<T>(BufferHeader.GetElementPointer(m_Buffer), index);
UnsafeUtility.WriteArrayElement<T>(BufferHeader.GetElementPointer(m_Buffer), index, value);

this bit here
BufferHeader.GetElementPointer(m_Buffer)

        {
            if (header->Pointer != null)
                return header->Pointer;

            return (byte*)(header + 1);
        }```

so if you write/read direct to it, it has to check this every call

which breaks simd

and just adds a huge amount of overhead

this is because it needs to check if the buffer exists in or outside the chunk

honestly, my preference would be that unity made buffers just always live outside the chunk

and use a FixedContainer or fixed[] if you want a chunk of memory to live in the chunk

[viral sonnet]

I was was already using AsNativeArray. Joachim pointed that out to me but it honestly didn't make any difference. Didn't know about the optimization in depth that can be done with it so it's odd I couldn't see any improvement

I've pasted this back then. Still the same read test

My buffer is a LOT bigger 😄

[rotund token]

im confused what you're comparing here

the lookup performance of a 250,000 buffer vs a hashmap?

oh wait no i see

[viral sonnet]

I just reduced the data to one integer and now DBs are a tid bit faster than NMHM

[rotund token]

wait im so confused

[viral sonnet]

I got slower results in any case with InternalCapacity(0)

in essence this does the same thing. iterate over all entities and check if a value is 34

one time data is in DBs, the other it's in a NMHM

[rotund token]

your buffer length is only 5 is that right

with 250,000 entities

[viral sonnet]

I allocate 5 elements for every entity, correct. So 250.000 * 5

[rotund token]

this test is fascinating but i think i know why the difference exists

let me test

[viral sonnet]

haha, "fascinating" - well this all started when I used a DynamicBuffer for cooldowns and calling Clear in a job took like 1.5ms for 250k

then it went down the rabbit hole

[rotund token]

just to confirm running it as is

i do probably get similar results to you

[viral sonnet]

they match pretty much. you have a faster cpu! 😄

[rotund token]

yeah i splashed out for a 3900X a couple of years ago

paired with my awesome 1060gtx 😐

on my ultrawide

is not a great gaming experience

but a great dev experience!

[viral sonnet]

nice! I'm still on an i7-4770k

[rotund token]

i had a 3570k before this

it was getting slow

[viral sonnet]

sooo many worker threads drool

[rotund token]

oh that screenshot

is only half of them 😐

[viral sonnet]

lol sick

[rotund token]

it actually becomes a pain to use the profiler

first world problems

[viral sonnet]

damn lol 😄

my NHM test has 2.75ms. 1ms slower than yours

[rotund token]

so having a looked at the bursted code

what i think the different is

isn't the actual nmhm vs hash

it's the overhead of the job

[viral sonnet]

that's what I've been saying! 🙂

[rotund token]

have you

all im trying to compare is apples

a native array on a component

vs a buffer

and you're like 7 levels deeper

what you seem to be comparing is
IJobFor vs IJobEntityBatch (or chunk in this case)

if you were storing a world grid on a single entity

its not going to be any different

but if you're saying the lookup time of 250k entities is a lot more expensive than iterating a native container

then i'm onboard

[viral sonnet]

Can you rephrase that? In my own words, the chunk iteration of DBs should be faster than a NMHM.

[rotund token]

entity iteration of chunks

is slow as entity count goes up

it has to loop through every chunk (even empty ones) and check the archetype vs the query

if you have a lot of archetype changes leaving empty chunks everywhere

[viral sonnet]

oh really, I wasn't aware of that

[rotund token]

you get quite a noticeable performance degeneration over time in a game

we have this issue a bit on old gen consoles

game might start at a solid 30fps, an hour later down to 24

[viral sonnet]

I thought this was a one time thing per frame. huh

[rotund token]

this is one of the reasons i've been moving to large jobs instead of lots of small ones

i believe 0.50 did a bunch of optimzations here though i havent benchmarked yet

[viral sonnet]

Hm, I've been doing the same thing although my archetypes are stable. No structural changes

[rotund token]

i think it does a lot more caching for this

especially if you aren't changing archetypes

so it might be less of an issue than it used to be

so take what im saying with a grain of sand

(about slow down over time)

i haven't benchmarked our console builds since we upgraded

[viral sonnet]

a downside is also that you get struct copies when reading DBs

[rotund token]

you dont have to

[viral sonnet]

in my old test I also get struct copies from the NMHM so that's fair but that can be optimized

[rotund token]

it has a
ref ElementAt(index)

what's missing is a nativearray version of this

    {
        public static ref T ElementAt<T>(this NativeArray<T> array, int index)
            where T : struct
        {
            return ref UnsafeUtility.ArrayElementAsRef<T>(array.GetUnsafePtr(), index);
        }

which of course i've added ^_^'

but yeah that should be built in

every other container has it now

db, nativelist etc

[viral sonnet]

right, let's see how it performs then

[gusty comet]

Is it generally a bad idea to instantiate prefab entities at runtime? I've heard about a push towards converting all entities at the beginning and avoiding creating new ones, even through prefab instantiation

[rotund token]

i fall on the side of it being bad

more for the future and 1.0 features which won't work from runtime created archetypes

but i also just think as a general principle in a larger team

things should be editable without code

[gusty comet]

Ok, so for entities that should only "exist" for short periods of time, do I toggle them through the Disabled component, instead of instantiating and destroying them on the fly?

[rotund token]

what type of entity are we talking

[gusty comet]

It's an entity that has a component ProximityComponent that is queried by ProximitySystem, but some of those entities will only exist for short periods of time

[rotund token]

(i should say there are definitely cases where entities can and should be created via code such as state entities etc just game entities shouldn't be)

yeah so that case seems fine to me

i think if it more like code
what should be public, what should be private

[gusty comet]

Could you elaborate on state vs game entities? It sounds like a useful litmus test

[rotund token]

if the entity should be changable by designers such as creatures, buildings, etc it should be Public therefore setup via conversion

[viral sonnet]

ElementAt makes zero difference in this test. That's so weird ...

[rotund token]

if an entity is for states used by developers only, etc it should be private and (usually) created on the spot

[rotund token]

yeah because that's not the performance issue on this test you aren't really iterating much

there are definitely cases where more developer based config should still be converted

like say, server settings etc

[gusty comet]

uh oh, the temporary ProximityComponents also has configurable TimeToLiveComponent values. IT's definitely a game/public entity. On the surface it's used for temporary proximity objects like incoming damage markers (a giant boulder is coming towards you)

Thanks for the tip

Though I'd argue that you can configure the instantiated entity if it's a prefab by editing the prefab's values

[rotund token]

so i guess you could break it into 3
public - designers/artists - conversion
internal - configuration - conversion but hidden away
private - application state without configuration - create in system

this is how i do it anyway

internal/private is more my preference than a hard rule i'd say

but at the very least i think anything designers/artists want to touch should be conversion

[gusty comet]

Thanks. I tend to see in black and white. I'll stick with instantiating prefabs since I'm just a one-man team and wrangling with Disabled is causing headaches

[rotund token]

personally all my settings exist on scriptable objects which map to a component during conversion

this way i can actually make changes to my settings/scriptable object, reimport subscene

and update the values at runtime (with writeback)

[viral sonnet]

tertle, do you have a suggestion for handling particles?

[rotund token]

nah not really i'm just putting this off

btw if you want to make your test faster

    public struct NHMElement : IBufferElementData```

take the buffer out of the chunk

so you can get more entities into it

[viral sonnet]

Guess me too for the 3rd time but I'm building a half-assed thing now 😄

[rotund token]

vs [InternalBufferCapacity(8)]

[viral sonnet]

huh, it was slower in my case with 0 cap. also my vs2022 can't decompile anymore, so no more integrated entities source code QQ

[rotund token]

small archetypes are more important than pretty much anything when it comes to a lot of entities

cache usage be damned

[viral sonnet]

yep, I need to redo a test where the chunk size is larger than 16k

[rotund token]

anyway that improvement makes me feel a bit better about my poor db

i really wnat to find a way to inject [InternalBufferCapacity(0)] onto LinkedEntityGroup

there is no reason that ever needs to live in the chunk =\

pretty sure i'm just going to be stuck editing that package at some point

but back onto new topic

i've been putting off figuring out my hybrid workflow

i kind of liked how my cinemachine wrapper worked though

so might just do something like that

not sure if it'll scale though

wish viual effect hurried up with a gameobject free implementation

[viral sonnet]

yep, just read today that it's postponed till 2022 with no more info

[viral sonnet]

For what reason?

[rotund token]

so it stops taking up 144 bytes

per entity

in my chunks

archetype size, 178

[viral sonnet]

it does even with cap 0?

[rotund token]

144 of that is a linked entity list 😐

[rotund token]

no with 0 size

it only takes up 16 bytes

[viral sonnet]

ah i see

yeah that's really problematic. Editor also injects lots of overhead comps

So you only know the final chunk capacity at runtime. or you calculate it

[rotund token]

well it is 2022 already! do you mean unity 2022?

[viral sonnet]

haven't checked if the new archetype window is more capable now

[rotund token]

ive looked at the visual effect script

i don't really see any reason it has to exist on a gameobject