archived-dots

[velvet panther]

Cheers. Any and all insights VERY welcome. I can't believe I'm saying this, but I think there's a need for a user generated wiki on the current state and best ways to use and exploit Burst and Jobs.

[coarse turtle]

think there was one started a while ago, but no one updates it - it's pinned in the channel

[velvet panther]

Problem One: very few people seem to be using Burst and Jobs, and most of them seem to be very serious folks with huge workloads and unable to spend time sharing back their insights. So we get lots of superficial fluff rather than profound insights.

Problem Two: there's a bit of a general misconception that Jobs and Burst are attached to ECS, whereas it's much more the other way around, and therefore Jobs and Burst should be heavily considered in terms of what they can offer now, to the MonoBehaviour users of the world.

[safe lintel]

hm - doubt anyone know why Tests would fail with blob assets while using them in non test code works fine?

[hollow sorrel]

not sure what kind of profound insights you want to see for jobs
jobs are just like C# tasks but with enforced data copying, it's meant to be "simple" multithreading where you don't have to worry about race conditions etc

so if there's a bulk of hidden profound information you'd need to use them properly, it'd be missing its goal

[safe lintel]

ok im a fool, was using AreSame when I needed AreEqual

[velvet panther]

#archived-dots message

I'd happily start with that simple insight.

What happens to GetSubArray? I don't need to dispose of it. But what costs are associated with using it, in terms of garbage, and supposed aliasing, even if it's not an aliased subset of the array?

If I use a NativeSlice like a reference, and constantly change its size as I need, what happens to the 40 bytes of the last one?

Since there's no struct element access, is it expected that I go low level and do byte level adjustments rather than making copies to edit them?

Is this a perfectly performant and accepted way to do struct element editing: https://docs.unity3d.com/ScriptReference/Unity.Collections.LowLevel.Unsafe.UnsafeUtility.WriteArrayElementWithStride.html

Why don't I have to dispose of Temp.Job allocations of NativeArrays?

How do I create a NativeArray without zeroing its contents?

Sorry if these are frequently asked questions. I've not been able to find this kind of information in a reliable place, nor somewhere that even seems to be current.

How much slower and less SIMD friendly is using a NativeArray<float>[] with four "elements" for convenient access than fussing with a NativeArray<float4> and injecting into its members with the above WriteArrayElementWithStride?

[willow plaza]

According to the docs it's "a view into the array that aliases the original array". In other words, it's like a pointer to the elements of the original array.

[willow plaza]

I assume that you can only change the view slice, so it doesn't really affect the original array nor allocates/disposes of anything in the memory.

[velvet panther]

@willow plaza it doesn't alias the array, it CAN alias the array, if I choose an offset that's not in alignment with the content's type. Does Unity forcibly mark as aliased, even if I'm careful to use SubArray settings in alignment with the type held?

[north bay]

Why don't I have to dispose of Temp.Job allocations of NativeArrays?
You have to dispose of TempJob allocations but not Temp allocations, temp memory is reset when the next frame starts
How do I create a NativeArray without zeroing its contents?
new NativeArray<>( length, allocator, NativeArrayOptions.UninitializedMemory )

[hollow sorrel]

Unity will never schedule jobs on main thread unless you use .Complete(), at which point it only potentially does because main thread would otherwise be stalled

[velvet panther]

How do you use/run a Job without Complete?

[north bay]

If I use a NativeSlice like a reference, and constantly change its size as I need, what happens to the 40 bytes of the last one?
Nothing happens to the last 40 bytes, they are still there, NativeSlice holds a pointer to the memory

[hollow sorrel]

usually you'd Complete at a time the job tends to actually be complete (e.g. next frame), or you can check .IsCompleted until it's done by itself

[velvet panther]

A NativeSlice is 40 bytes, created and returned when you set the size, regardless of whether or not it previously existed. So where does that 40 bytes go when I change the size and am issued a new NativeSlice?

[hollow sorrel]

btw GetSubArray, NativeArrays and NativeSlices are structs and don't allocate anything that needs to be garbage collected

[velvet panther]

The 40 bytes ISA native slice. It's a heavy struct. You get a new one when you adjust the size of an existing one.

Yes, they're structs, but they're held in memory somewhere, not on the stack, so they have to be collected at some point, right?

[hollow sorrel]

yes they're held in memory and not always on the stack, but that doesn't mean they have to be garbage collected

they're value types so C# doesn't reference count them

they're just held by whatever is using it and are gone when no longer held

[velvet panther]

So I can just create LOTS of nativeSlices, in a single frame (for convenience) and not worry about doing this a LOT!

I mean, this quite seriously, NativeSlices are a bit of a HUGE relief to thinking and coding for sub selection editing of an array, but am afraid to lean on them.

[hollow sorrel]

yea if you need to then sure
the whole point of all the NativeX collection classes is to specifically avoid allocating garbage

[velvet panther]

Cheers, this is VERY helpful and confidence inspiring, and what I'd consider to be wisdom, and I understand this to be their intent, but has anyone tested to make sure? I've used Unity for long enough to doubt ANYTHING that they claim.

[north bay]

Since there's no struct element access, is it expected that I go low level and do byte level adjustments rather than making copies to edit them?
If you don't like having to make a copy you can use an extensions like this:

 
        public static ref T AsRef<T>(this NativeArray<T> array, int index) where T : unmanaged
        {
#if ENABLE_UNITY_COLLECTIONS_CHECKS
            if (index < 0 || index >= array.Length) 
                throw new ArgumentOutOfRangeException(array.Length + "  " + index);
#endif
            unsafe
            {
                return ref UnsafeUtility.ArrayElementAsRef<T>(array.GetUnsafePtr(), index);
            }
        }

I think most people use a copy and assign it back

[hollow sorrel]

yea I mean the profiler tells you whenever you're allocating any garbage
but even if you don't trust that you can just go check how they were implemented, the code is visible
it's all implemented in C# except for the allocating data part which is native, because it's using native memory allocation instead of C# garbage allocated memory

[north bay]

But I always use AsRef extensions for native collections

[velvet panther]

@north bay I'm already doing a very light hack into the array, in an unsafe way, and it's working, but it's a lot of work to do it this way. And it feels "dirty" to have to work so hard to get struct element editing

[velvet panther]

I've read it all, I've done my level of testing (not great, but enough to show I don't know what's going on with the 40 bytes each time) and still don't trust it, because it's Unity, and Unity's ALWAYS got caveats for the unwary.

[hollow sorrel]

not sure which 40 bytes you're referring to

[velvet panther]

The sad part, direct memory access to the struct elements is 4x faster, at minimum.

[hollow sorrel]

if you're talking about profiler showing it's allocating 40 bytes, that's probably because you're in editor and have leak detection turned on

[velvet panther]

NativeSlice struct = 40 bytes.

No, it's just allocating it, I know, because of reading the source.

And I know it's being created, a new struct, each time I edit the NativeSlice. I'm just not sure if I do this 50 times in a frame, what's happening to that 50 x 40 bytes of stack used, memory stored structs

It might be that they're being stored somewhere that's getting dirtied, and after a while "fills" up. It's not clear. Nor is it clear why SubArray must be aliased, or it it's aware that it might not be aliased.

I'm doing a bunch of DSP with audio, that has to happen fast, to get that relentlessly hungry audio buffer filled, hence the need for speed and surety.

[hollow sorrel]

oh that's what you mean

yea the struct size is 40 bytes
but you don't need to trust unity for this
structs not being garbage collected is a C# thing and you can look into microsoft's docs if you wanna know more how structs work

[velvet panther]

I know how structs work, but these aren't just any structs, and we can't really see where they're storing them. They are storing them. They're not just on the stack, they're going to memory, somewhere.

[hollow sorrel]

they are just any struct tho
the only fancy part is that they hold a pointer to native allocated memory

[north bay]

    public unsafe NativeSlice(NativeArray<T> array, int start, int length)
    {
      if (start < 0)
        throw new ArgumentOutOfRangeException(nameof (start), string.Format("Slice start {0} < 0.", (object) start));
      if (length < 0)
        throw new ArgumentOutOfRangeException(nameof (length), string.Format("Slice length {0} < 0.", (object) length));
      if (start + length > array.Length)
        throw new ArgumentException(string.Format("Slice start + length ({0}) range must be <= array.Length ({1})", (object) (start + length), (object) array.Length));
      if ((array.m_MinIndex != 0 || array.m_MaxIndex != array.m_Length - 1) && (start < array.m_MinIndex || array.m_MaxIndex < start || array.m_MaxIndex < start + length - 1))
        throw new ArgumentException("Slice may not be used on a restricted range array", nameof (array));
      this.m_MinIndex = 0;
      this.m_MaxIndex = length - 1;
      this.m_Safety = array.m_Safety;
      this.m_Stride = UnsafeUtility.SizeOf<T>();
      this.m_Buffer = (byte*) ((IntPtr) array.m_Buffer + this.m_Stride * start);
      this.m_Length = length;
    }```

There's really nothing special about NativeSlices

[velvet panther]

50 x 40 bytes for a low latency audio processing routine = 2kb of "garbage", and this is happening at 256 byte audio buffering rates, so 172 times per second, which builds up to 384 Kb per second. And I'm wanting to do this throughout the game.

[north bay]

Well you aren't going to keep all that memory alive right?

[hollow sorrel]

that's not the same thing as garbage in C# context

[velvet panther]

I don't have any say in the NativeSlice structs. Where are they being stored by Unity?

[hollow sorrel]

you are storing them

they are just holding a pointer to whatever native memory you're slicing a view of

when you're doing NativeSlice thing = .. that's you storing that NativeSlice right there

[velvet panther]

That storing of the pointer is a struct, it's 40 bytes in size. Each time I change the values in the struct, I'm getting a new struct. What happens to the old 40 bytes?

[hollow sorrel]

if it runs out of scope it's gone forever

only reason it'd still be there is if you're storing it as a member of your class or in a different array somewhere for whatever reason

[velvet panther]

The NativeSlices, I have one, I change its size 50 times in a frame of editing. Not what it's pointing to, it's always pointing to the same NativeArray. I change its ranges 50 times. That's 50 new 40 byte structs. This is safely handled?

By that, I mean "can I be absolutely sure I

I'm not generating garbage?

[north bay]

You aren't going to generate garbage
You should look up how stack allocations work in C#

[velvet panther]

These aren't merely C# stack allocations, and we're living in a world where Unity is using C# as a scripting language. They're holding the structs somewhere special, I presume, because i can keep them around, in a location, by reference. NativeSlice and NativeArray, whilst structs, act much more like reference types in their own right. Do you see what I'm getting at?

It's not like a Vector3, a NativeSlice is a carefully managed, special kind of struct, that Unity is treating much like it might do a Reference Type.

[coarse turtle]

a NativeSlice is just a struct with a pointer that points to content

still done by value

[velvet panther]

If they're (NativeSlices) truly just structs, and garbage free, why do Unity provide so many low level unsafe operations seemingly to prevent the need to recreate or create lots of them?

[coarse turtle]

the usage of a NativeSlice is to point at already existing memory

so you can create as many as you want - based on their implemented constructors

[willow plaza]

Can you elaborate what you mean by low level unsafe operations?

[velvet panther]

https://docs.unity3d.com/ScriptReference/Unity.Collections.LowLevel.Unsafe.UnsafeUtility.html all these, plus the special few for NativeSlice.

They look, to me, like they Unity doesn't trust NativeSlice nor GetSubArray, and is therefore almost suggesting you "roll your own" direct access ways of managing sub selections of NativeArrays.

eg. Animation Legacy component is left in Unity. Once you realise just how incredibly suboptimal Animators are, you understand why Animation Legacy is NEVER getting removed. When you find out it's nearly impossible to successfully cache/pool with Animators, you wonder why this isn't explained somewhere significant. When you realise how unreliable the state machines are for Animator Controllers, you realise you're almost always better off rolling your own and working with Animation Legacy. - this is a normal learning cycle with Unity systems. 3 levels of caveats. When I look at all that low level stuff, and experiment with a bit of it and discover that some of it is INCREDIBLY fast (as Animation Legacy is to Animator) I begin to think I should just ONLY use the low level stuff and NativeArray as a convenience start point..

memcpy, for example, is BLINDINGLY fast, once you make your usage of it that avoids ALL safety checks. So fast that I was, at one point, overwriting them with older series of unique data 100+ times in a portion of a frame, across a dozen arrays, and wasn't even noticing a single hitch... (due to a needless while loop that'd been left in by my mistake). CPU usage increased by about 25% for that portion of the AudioThread, but it was from 3.9% to 5%... so I thought this was just system noise. Cleaned up this incredibly wasteful loop, that was probably copying about 200,000 floats into quite fussy sub portions of a dozen arrays, within very particularly conditionals that determine what to copy, and the CPU usage dropped back to exactly 3.9% on that thread. I think my conditional matrix was probably half of that 25% increase of CPU usage. memcpy is so fast I can't even really comprehend it.

*keep in mind, by "frame" I mean 256 byte audio buffer = 1/172nd of a second. And I'm testing on the oldest Macbook Pro I have, which is OLD, and when it's throttled for heat, so that I can see it struggle.

[foggy drum]

how can i detect a "OnCollisionEnter" event? im currently using ICollisionEventsJob and tried to set the CollisionResponse flag to none of the collider, but its more like a one-shot thing

[safe lintel]

@foggy drum I havent personally used them, and iirc they only exist in the samples repo because they arent yet part of the actual package but here https://github.com/Unity-Technologies/EntityComponentSystemSamples/tree/master/UnityPhysicsSamples/Assets/Demos/2. Setup/2d. Events/2d2. Collision Events

[foggy drum]

thanks, i'll try it out and let u know if it works

[twin raven]

I am checking raycast from mouse to world in a system Update function, and it is running normally. But if I have a code to set a singleton, the system won't be ran at all 🤔

SetSingleton(new EditTarget { Entity = e });

Is there some automatic logic that prevents system from running since there is no singleton of that type to begin with?

Maybe I should change the logic to tag the entity if it is being edited, but nevertheless it is not clear that what condition stops the system from running.

[coarse turtle]

SetSingleton(...) internally creates a tracked EntityQuery with the component you're looking for. The EntityQuery gets tracked as an EntityQuery dependency, which is why your system won't run, if there is no singleton entity available

[devout prairie]

Anybody familiar with this error:

Basically it's only happening when i increase the number of entities i'm spawning and running through my system

Could it be related to the fact that i have around five ForEach jobs linked together by dependency, with one ECB passed from beginning to end?

[safe lintel]

did you also AddJobHandleForProducer for each job where the ecb is used?

[devout prairie]

well not for each job, basically at the end of the chain of ForEach jobs i call addjobhandle and pass in the Dependency

[safe lintel]

think you need it for each job where the ecb is used, right after you schedule then addjobhandleforproducer

[devout prairie]

So basically like this:

var handle1 = ForEach().ScheduleParallel(Dependency);
var handle2 = ForEach().ScheduleParallel(handle1 );
var handle3 = ForEach().ScheduleParallel(handle2 );
Dependency = ForEach().ScheduleParallel(handle3 );
endSimECBSys.AddJobHandleForProducer(Dependency);

something like that

and the ecb is passed in to the two jobs it's actually used in using WithNativeDisableParallelForRestriction(ecb)

so it's working fine, until i change from say 20 spawns to a higher multiple

i don't think it's a problem with my code every seems watertight

and as all jobs are chained together i thought, i can't see where it could possible invalidate something structurally

sorry just thinking out loud here hehe

[safe lintel]

pretty sure you need to use it for each job, also you could simplify(well sorta) that to

Dependency = ForEach().ScheduleParallel(Dependency);
endSimECBSys.AddJobHandleForProducer(Dependency);
Dependency = ForEach().ScheduleParallel(Dependency );
endSimECBSys.AddJobHandleForProducer(Dependency);
Dependency = ForEach().ScheduleParallel(Dependency );
endSimECBSys.AddJobHandleForProducer(Dependency);
Dependency = ForEach().ScheduleParallel(Dependency );
endSimECBSys.AddJobHandleForProducer(Dependency);

assuming each of those foreach's actually write to the command buffer

`Remarks

When you write to a command buffer from a Job, you must add the of that Job to this buffer system's dependency list by calling this function. Otherwise, the buffer system could execute the commands currently in the command buffer while the writing Job is still in progress.
`

[devout prairie]

hmm.. the only issue is i kinda NEED the jobs to be daisy-chained like that otherwise i think they'll be scheduled potentially out of sync

[safe lintel]

that is daisy chaining em 🙂

[devout prairie]

i use the ecb if two out of the five or whatever jobs, maybe i'd be better trying just create two ecb's then

[devout prairie]

ah yeah, so it is!

hmm

i'll try that now thanks

[safe lintel]

well in any case if im wrong, its not a big code change

[devout prairie]

hmm sadly not

i'll try two different ecbs

[foggy drum]

it seems to be working fine 👍

[devout prairie]

for what it's worth, these are my first three chained jobs, with your suggestion:

[safe lintel]

pretty much how I do things but sorry it didnt work

[devout prairie]

hmm two separate ecb's as an alternative didn't work either, separate error for that

just weird that it works perfectly until i use a greater amount

will need to painstakingly investigate

[devout prairie]

No known issues in general with using an ecb in a ScheduleParallel with WithNativeDisableParallelForRestriction no?

[tight blade]

anyone know how to get information about gravity from the physics system from a custom system?

I kind of just want to know what gravity's strength and direction is

feels like that's probably a singleton-like thing somewhere in the physics systems

it is! its the PhysicsStep singleton component!

[digital kestrel]

do you guys use subscenes? my authoring scripts always fail to run when the game object is in a subscene

[left oak]

I assume you mean besides this: "Parallel access is only safe when each thread only accesses its own, unique range of elements in the container. If more than one thread accesses the same element a race condition is created in which the timing of the access changes the result."

[velvet panther]

If I make a custom struct with (let's say) a couple of floats and ints as elements, and throw a few of these structs in a NativeArray, in normal code, I can individually edit the values of the elements eg: myStruct[1].myFloat = 20.25 without any problem. Once I put this in a Job, I can't do this kind of access. Why is this?

[vital sandal]

You need to take the value out, modify it and return it to the array

like:

MyStruct struct = myArray[index];

struct.value = newValue;

myArray[Index] = struct;

[velvet panther]

@vital sandal I know what I have to do. I'm asking why I have to do that in the context of Jobs.

[vital sandal]

oh I read that .... mmmm

I mean I was on that page moments ago

here

Due to the lack of ref returns, it is not possible to directly change the content of a NativeContainer. For example, nativeArray[0]++; is the same as writing var temp = nativeArray[0]; temp++; which does not update the value in nativeArray.

https://docs.unity3d.com/Manual/JobSystemTroubleshooting.html

[velvet panther]

Yes, I've read that - it doesn't explain why. I've also watched the video from 2018 where they twice said they'd soon provide element level direct access. As I say - why isn't it possible?

[vital sandal]

Well it does explain why.

"Due to the lack of ref returns"

[velvet panther]

No, it's just another way of describing the problem. It is not an explanation of why the problem exists.

Why is there a lack of ref returns?

Why, when they specifically said, in 2018, that they were addressing this missing functionality, is it still not possible to do direct element editing?

[vital sandal]

Well... cause it's Unity?

[velvet panther]

Good point. Their ratio of promises made to promises kept is a bit unique.

[vital sandal]

They know what they have to do but they lack focus.

[velvet panther]

Of course, the use of "promises" is hyperbolic. Take it with a grain of salt. I'm just very surprised they haven't made this possible when all the tools to do it are there, already. I've made my own, using Unsafe Utilities, and it's FREAKY how fast it is.

[vital sandal]

Maybe you should should send in a pull request XD

[velvet panther]

You kidder 😉

[vital sandal]

So how faster is it?

[velvet panther]

At the slowest, with a float4, it's 4 to 5x faster. With a custom struct, it's anywhere upto 20x faster.

[vital sandal]

I want to know more

[velvet panther]

And I'm using a small custom struct. So a big custom struct it would be MUCH faster.

[vital sandal]

mmm

[velvet panther]

I haven't even updated my little hack to use memcpywithStride or whatever it's called. Which could potentially make it hugely faster, given how fast memcpy is.

[vital sandal]

Well you can send me a pull request, I'd sure like to take a look at that

[velvet panther]

The tricky part is getting a ptr to the struct, then recasting that to the type you want to access, then offsetting the pointer to exactly that spot in the struct.

The code is super basic..... hang on... lemme find one of my little runners.

[vital sandal]

Ya see that's the part I'm not familiar with. Pointers are my next level­ to climb...

[velvet panther]

how do I insert code? Not a discord champ.

[vital sandal]

Hahahahahahaha

I know what you mean

took me a while to figure it out

[velvet panther]

Surely I don't screenshot it, right?

[vital sandal]

you need to put 3 tick marks

[velvet panther]

those

[vital sandal]

`

[velvet panther]

argh. ok.

[vital sandal]

that -> `

[velvet panther]

        j = overlapping;
        for ( int i = attakFinish ; i < noizeLength ; i++, j++, ptr += 4 ) {
          *ptr = dekayCrv.Evaluate( j * stepDekay) ;
        }```

+= nID = the offset to the desired float in a float4

[vital sandal]

what is float*

[velvet panther]

That's a pointer to a float

[vital sandal]

oh

[velvet panther]

float4* is a pointer to a float4

*pointerName is how you access the memory at that float's position in memory.

[vital sandal]

so it declares you are creating that type of pointer?

[velvet panther]

Yes,

[vital sandal]

oh ok

and ptr is a keyword?

[velvet panther]

void* is kind of the generic pointer "type", whereas the specific pointer types like float* are aware of their own length in bytes

[vital sandal]

oh ok

[velvet panther]

ptr is just my variable name for the pointer. Sorry, very lazy, compact naming used.

[vital sandal]

oh ok that makes sense

well it's not you

[velvet panther]

ptr += 4 means jump 4 pointer locations (across the float4) to the next element of the same location in the next float4

[vital sandal]

it's me I havent had to use pointers before and never touched c++ so the concept is still a little abstract in my mind

[velvet panther]

argh, I grew up on C. Never used C++. So pointers make a bit of sense to me. But the way to messily access them in HPC# is a bit... funky.

[vital sandal]

It's like knowing what something does and is but not understanding the implications yet

[velvet panther]

pointers are the tangible memory address where things are stored. So they're "physical" in nature, which is much easier to think about, for me, than the abstractions of OOP.

[vital sandal]

Oh it points to a memory allocation in ram and you can reuse that pointer to access that allocation

[velvet panther]

YES!

You got it. So you can put anything in a location, and go straight to it, every single time. This is why they're so fast.

[vital sandal]

that I know but... the syntax is still problematic

I tried looking for a good tutorial on it... didnt find any that didnt came with aheadache

[velvet panther]

Yes, the syntax is MADNESS if you're coming from OOP. It was madness going to OOP, for me.

[vital sandal]

I bet

[velvet panther]

Ask me any questions you might have, it will be an interesting challenge to see if I can decompress what I know.

[vital sandal]

I guess I will bookmark that code and come back to it

[velvet panther]

I have done about a thousand tests during the last week or so, to figure out what low level HCP# is doing and can do. It's the first time I've enjoyed programming since I started using Unity.

[vital sandal]

well this part here ptr = (float*)(p + attakFinish); ptr += nID;

[velvet panther]

Sorry. this is missing VERY important context.

hang on... let me find the line that will make this clearer...

[vital sandal]

and ptr += 4

the rest looks pretty normal

[velvet panther]

var p = (float4*)noizADSR.GetUnsafePtr (  );````

[vital sandal]

oh

[velvet panther]

noizADSR is an array of float4

p is a pointer to the very first float4 in that array

[vital sandal]

and when you do (float4*) you are casting the pointer to a type or declaring it?

[velvet panther]

GetUnsafePtr is one of the unsafe operations Unity provides

Yes, you can either do a generic (void*) pointer, or set it to what it actually is, a (float4), so that it knows that it's 16 bytes long

a float4 is 4x 4bytes of floats, so 16 bytes

[vital sandal]

and that start in (float4*) ok the the star has no other meaning that saying you mean a pointer

[velvet panther]

Yes, it's a very old convention, that should have been replaced by something like (floatPtr) by now.

I'd prefer it was (ptr_float4) or something like that, so I'm reading the fact it's pointer first.

[vital sandal]

so in ptr = (float*)(p + attakFinish); you are assigning a pointer of type (float*) to a pointer variable called ptr

and that pointer is p + attackFinish?

wasnt P apointer?

[velvet panther]

p is a pointer to a full float4, so 16 bytes long. ptr is a float point, so only 4 bytes long, and I've offset it to nID, which is either 0, 1, 2 or 3, so that it's x, y, z or w of the float4

Each float4 inherently has 5 pointers. One to itself (the p in this case) and the four internal floats.

[vital sandal]

Ohhhhh

thast whats going on

so you offset it after setting its value?

man I need to find a a tutorial on this

[velvet panther]

Yes, I set the p to the very first pointer to the very first float 4, then create and offset my ptr to the element I want to iterate through, and then iterate by stepping 4 x 4 bytes (ptr += 4) to go through the array at (say, location y)

[vital sandal]

It's just that finding one from someone who speaks clearly is harddddd.... if you know what i mean

[velvet panther]

I don't think there is one. I think I'm going to have to try write it up, for you and everyone. The speed is astonishing.

[vital sandal]

OH ok so P is a reference to the start of the memory block

[velvet panther]

I had to discover all of this, and go back and forth through a thousand experiments to figure out how to do it

[vital sandal]

and ptr is the pointer you offset around to get access to your values

[velvet panther]

Yes, it's both the start of the first memory block AND the first float4 in the array.

[vital sandal]

obviously cause array[0] just starts right there

its raw memory

not a type

[velvet panther]

And ptr is then created at that location, and multiplied by the offset value of 0 (first one) or 1, 2 or 3 to get at the element offset desired, depending on whether I want to edit x, y, z or w

[vital sandal]

It's a god damn flat array

[velvet panther]

YES!!!

You got it!!!

[vital sandal]

haha had to f arround with those this week... was a 1st...

[velvet panther]

texture?

Were you messing around with the getrawdata of a texture and then changing pixels? I had that fun a couple of weeks ago.

[vital sandal]

haha no, I'm using it to send a multilevel array containing loddata to a job

[velvet panther]

strewth. Way more complex!!!

[vital sandal]

well apparently you cant have an array in a struct and pass it on to a job.... unless you know how to mes arround with pointers

[velvet panther]

If I write up a little introduction to pointers, pointer semantics, pointer syntax and their speed/benefits, where should I put it?

[vital sandal]

On a bkig i guess?

i mean blog

[velvet panther]

That's another good point, arrays in structs are a messy issue. I've avoided it. But now that I have my pointer access working, will probably start putting arrays in structs.

[vital sandal]

any free website / blog place

yes hold on

someone gave me the recipe i just didnt understand it all back then

[velvet panther]

Is there somewhere people tend to gravitate to for Unity blogs? Like Medium is popular for iOS design

[vital sandal]

let me find it

most things like that I stumble on are on blogs

oh yes hold on i know a place

check this place https://www.gamasutra.com/

https://www.gamasutra.com/

[velvet panther]

argh, yes. That place... that's become a bit of a ghost town over the last few years.

[vital sandal]

yeah but I havent found anything that replaces it.

Only other thing I've seen are personal websites type of thing

[velvet panther]

Weird how few people write about programming in Unity, well. JacksonDunstan, CatLikeCoding...and I think we're done.

[vital sandal]

You can probably post something in the forums

theres a lot of good stuff on youtube

that guy makes excellent stuff https://www.youtube.com/channel/UCmtyQOKKmrMVaKuRXz02jbQ

full tutorials on very complex systems

[velvet panther]

He was good. I think he's left Unity for Unreal, or something else.

[vital sandal]

ItsJustBlank wrote to me the other day: "A custom struct with a T* ptr , length would be enough as you can override the bracket operator. You might also need[NativeDisableUnsafePtrRestriction] if you go down this route you will end up with a basic struct wrapper that can be treated as an array"

concerning arrays in a job

or rather in a struct that you send to a job

[velvet panther]

That makes a bit of sense. It's nested stepping, inside the array inside the struct.

In your lingo, it's a flat array, with an array inline in that flat array.

[vital sandal]

Ohhh ok

well that was an interesting discussion, do let me know if you release anything ... now I gotta go beat the crap out of my nativearray until a figure out why I set its value in a job once and then it stops changing the value.... even though it is setting it.

big fun, I wonder if I stumbled on a bug or something, wouldnt be the first time unity did that to me

[velvet panther]

Unity and bugs.... Oof. You an Aussie?

"crap"

[vital sandal]

Wrong side of the planet

[velvet panther]

Pom?

[vital sandal]

CA

[velvet panther]

argh. Canucks and their seasons 😉

Cue Daniel Tosh joke about seasons.

[vital sandal]

😁

[velvet panther]

If you're array is not getting set, see if what you're doing is another set, right at the end, that's the one you did at the beginning. This was a bug of MINE (those are rare 😉 ) that I struggled to catch because the Job was so fast I wasn't seeing a performance hit for iterating over the array 20x

[vital sandal]

what

you mean like the previous job didnt release the nativearray fast enough?

Oh n look at that

I fixed it while we were talking

haha

[velvet panther]

No, it was me. I was setting the values, then doing a bunch of maths and iterating over all the values and adding/multiplying them, and then, somehow, doing the the initial set again, right at the end, overwriting all the work. A

[vital sandal]

I just didnt test it yet

must have hit ctrl-z one to many times and lost track of what got reverted

I hate when that happens

[velvet panther]

I initially thought the work wasn't getting done, as it was happening so fast that's what it seemed like was happening. but it was me overwriting the work.

[vital sandal]

oh yeah that happens too

Do you mess around with compute shaders?

check this out https://www.sebaslab.com/the-1millioncubes-challenge-how-i-managed-to-animate-a-freakload-of-cubes-on-windows-using-unity/

[velvet panther]

No, I'm flogging the death out of the graphics system already with too many particles and blends, so am using Jobs to exploit what's left in the unused cores of the CPU and the AudioThread.

[vital sandal]

hehe

[velvet panther]

btw, if you ever need to do stuff, the AudioThread is a dark little corner in which you can do things off the main thread. Just use OnAudioFilterRead, it's a Unity Engine Callback like "Update" but on a unique thread. Throw it in a MonoBehaviour, add an AudioSource to the Object and you can abuse it for whatever you want.

[vital sandal]

Hes using some gpu feature which appears only supported by Vulkan... sucks

really?

[velvet panther]

Yes, compute is quite limited in support, mostly DirectX

Yes, the Audio thread is a little wonder!!!

[vital sandal]

So anything you put in that event is runned on a separate thread.... I guess you still have to be cautious about data access

[velvet panther]

Yes, just use NativeArray's and you'll be fine. Lock them if you want to be pedantic. I don't bother. Just use booleans to say when work is or is not done.

[vital sandal]

Oh ok

Thats good to know

But with the job system why bother?

[velvet panther]

The AudioThread is called into by this OnAudioFilterRead at the rate of the Audio System, which defaults to 1024/44100 per second on a Mac/iOS and 1024/48000 on PC

Because there's somethings that you might want to do light and fast, and still use Jobs for longer things. this way you don't have to mess with queuing for long (multiple frames) jobs versus short "jobs" you can do on the AudioThread

So you can make your own little microJobs queue system on the Audio Thread, which is what I'm abusing it for.

[vital sandal]

but 1024/44100 thats what every 0.02 seconds?

[velvet panther]

Yes, it's an odd "frame rate". I set mine to 256/44100 because my game is VERY fast.

256, 512 and 1024 are the audio buffer sizes. How much audio data is required to render a "frame" of audio.

[vital sandal]

so thats like running in update at 100fps

[velvet panther]

at 256/44100 it's 1/172nd of a second

[vital sandal]

Well when your done posting your blog make a post on your game 🙂

Curious what you use all that juice for

[velvet panther]

Will do. It's a lightning fast pinball meets sonic type of game.

[vital sandal]

ohhh niceeee

hey you wouldnt happen to know what Array Aliasing is?

[velvet panther]

Yes, as a matter of fact... I do!

One of the few things I do understand 😉

[vital sandal]

I was looking at [noalias]

not simthe atrribute ...

https://docs.unity3d.com/Packages/com.unity.burst@1.4/manual/docs/OptimizationGuidelines-Aliasing.html

not sure I get it is that talking about the index or ?

Im sure its not

[velvet panther]

If you do things in a sensible way (with regards to setting up and iterating through arrays) you should not have to worry about this, you should be getting the benefits of no aliasing automatically.

[vital sandal]

even if I'm using a flat arrays?

[velvet panther]

I don't think you can mess up when using flat arrays, to make aliasing an issue. You'd have to do something silly.

[vital sandal]

well obviously I dont understand the implications

It came up while looking into Loop Vectorization

[velvet panther]

Yes, this is the bit I'll need to explain. Remember the float4's... they're a very good example of this, in both ways. How aliasing can be an issue, and how when theres no aliasing there's HUGE performance gains with Burst.

Yes, you're onto it. I think their verbiage is narcissistic nonsense. They should just say "wide is good, so go deep on wides" and start explaining float4

[vital sandal]

ok but hwo do you cause aliasaing?

[velvet panther]

The simplest and stupidest aliasing would be what I'm doing, injecting floats into float4's. This isn't technically aliasing, but it's similar in that I'm loading a float4 into the registry, and then a float into another registry, thereby not using 3 of the 4 slots in that second registry, and wasting a chance to do 3 other things at that same time.

register, sorry. Autocorrect hurting me.

The ABSOLUTE ideal is that you do float4 + float 4, as this uses the full width of the average register, and is SUPER fast for processors to do. Anything that's a portion of this is "aliasing" to an extent.

[vital sandal]

ohhhh ok

as if you are using float4.x and .y only

[velvet panther]

The really bad aliasing is when you seek to read a value that's going to change in the same cycle, so the processor gets all fuddled and goes back and forth trying to figure out state it can safely read, gives up and uses an external memory spot to store a result, and then retrieve that to do a set

[vital sandal]

oh ok so instead of doing a+a b+b c+c d+d you do float4(a, b, c, d) + float4(a, b, c, d) and its super fast cause the operation is assingned to a cpu function

or rather a dedicated cpu operation just for float4 additions

[velvet panther]

That's somewhat ok. But if you do: float4(a, b, c, e) * float(a, b-a, c+=e, e) you're frying the thing.

[vital sandal]

yeah that was my understanding

so you would have to prepare your values first then do all your additions

[velvet panther]

YES! It's a bit of work, but the easiest way to avoid aliasing is to make unique arrays for each input and have another unique array for outputs, and then try to make sure they all have exactly the same types, and very similar index values.

[vital sandal]

or multiplicatinos

well thats why I thought a flat array could cause issues

[velvet panther]

If you can organise your data so that it's arrayOne[i] * arrayTwo[i] + arrayThree[i] etc, this is SUPER fast.

[vital sandal]

yes but only with arrays of the same length

[velvet panther]

Break up your work into the bits that overlap. It seems like a lot of work to streamline the data, but memcpy is SUPER fast for doing this, and the benefits in terms of processing speed are UNBELIEVABLE so it makes up for the effort to streamline and sort data.

[vital sandal]

ie.: my subarrays are 100k index offset (one for each LOD level so 200k index) but my position array is only 100k

[velvet panther]

All those claims about how fast Burst are, they're all coming from having streamlined the data, absolutely perfectly. Unity is kind of cheating in all their examples as they use ABSOLUTELY PERFECTED sorted data to show the gains.

[vital sandal]

ive been trying to do this but my multilevel arrays are messing it up

[velvet panther]

In your situation, align the portions of the arrays that can use the same index values, and do that as one Job, and then do a leftovers run where you have trouble aligning the index values.

I call this "portioning", but I think others call it chunking.

[vital sandal]

I could use all arrays of the same length but I would have to hard code 2 array for each lod level which would hard code the number of possible lodlevels

so a job for each lod level

[velvet panther]

And don't be afraid to use multiple indexes with offsets, like in my pointers example, where i'm incrementing i, j and ptr

[vital sandal]

well thats what the flat array is for

that im using

[velvet panther]

Make more flat arrays, that's the easiest way to make it streamlined.

[vital sandal]

then I would double the amount of data

[velvet panther]

If it's too messy to use multiple indexes, make more arrays. Using more data is CHEAP!!!

It's amazing how fast it can predictively pull in data from several arrays, and they can be temporary creations, and fast, too. Use memcpy to flesh them out. It's so fast it'll blow your mind. You have to measure in ticks, even with tens of thousands of structs.

[vital sandal]

well Ive been using GetSubArray to send chunks to the jobs

[velvet panther]

I had a choice between using GetSubArray, NativeSlices, or just making more arrays and creating multiple indexes. I chose the latter, and despite having a bit more code to sort out the ranges and align things for final iterations, the speed is astonishing.

[vital sandal]

but you cant.... send a chunk of a flat array

[velvet panther]

This is what I use memcpy for... I create new arrays that exactly sized as I want, same as you're using GetSubArray, but new arrays.

[vital sandal]

I tried a method that uses memcpy getsubarray seemed faster

but it may be in my head

[velvet panther]

memcpy is unsafe, but it's able to do portions of an array to portions of another array, as copies. So I create buffers bigger than I'll ever need, and then use memcpy and ranges to create the portions i need.

getSubArray is faster, it's only a pointer to a portion, but if there's overlap between your subarrays, you'll get aliasing.

[vital sandal]

thats probably why, from what I understand with GetSubArray there is no copying going on

oh ok

at least i got that right haha

[velvet panther]

Yes, that's right. GetSubArray is just a manual ranger.

I don't do much memcpy from frame to frame, it's all done before hand, in prep phases.

[vital sandal]

I wish there was a way to map a struct in a way that you can use it in an array but get a member of all the structs in the array as an array of all those members

well

ok that made no sense

[velvet panther]

Yes, sorry. I read that 5x and was more confused afterwards 😉

[vital sandal]

basicly you could use myArrayOfStruct[index] but also myArrayOfStruct.position[0] were position would be an array of all the "position" form all the struct

well I mean myArrayOfStruct.position[index]

[velvet panther]

Argh, this is STRIDE!

[vital sandal]

are you KIDING ME!

[velvet panther]

Stride is the number of steps to the data you want inside an array inside an array (or struct inside an array of structs) etc.

[vital sandal]

ok im lost

[velvet panther]

Once you've figured out the length of the stride from one location to another, you can do a memcpyWithStride (I might have the same wrong) or readWithStride, which is stepping to these locations, UNSAFE, but exactly as you describe.

[vital sandal]

but you have to make a custom collection

[velvet panther]

Primitively, you'd be doing this: a pointerArrayOfStruct to the struct you want to read or write to, and then stride into that pointer at the resolution and position of the "subPointer" to the exact element you want to read or write.

[vital sandal]

how fast is it? it must still be slower than making individual arrays

but how much

[velvet panther]

No, what I'm describing you do ontop of what you've created. Can you give me an example of your array and the struct and the element you want to edit/read and I'll get more specific.

[vital sandal]

Oh ok so same concept as the flat array again but using a pointer instead like the thing you are doing to access elements directly in your struct

[velvet panther]

Yes, ironically, how this conversation begun is exactly what you need.

[vital sandal]

hahaha

Ok this is the job I create ```manager.renderModuleJob = new MatrixUpdateJob { // Init the job
positionData = manager.positionData.GetSubArray(indexRange.min, indexRange.length),
rotationData = manager.rotationData.GetSubArray(indexRange.min, indexRange.length),
startIndexes = manager.flatNativeArrayConfig.startIndexes,
minIndex = indexRange.min,
renderOffsetData = manager.flatRenderOffsetData,
scaleData = manager.flatScaleData,
lodData = manager.lodData.GetSubArray(indexRange.min, indexRange.length),
LODLevelCount = manager.entityDefinition.gpuiLOD.Length,
outputArray = manager.flatMatrix4x4Data,

            }.Schedule(indexRange.length, batchSize, manager.lodModuleJob);```

[velvet panther]

You'll just be doing a slightly more complex version of what i was describing with float inside float4, wherein it'll be float inside struct inside array of structs.

[vital sandal]

and this is the job ```public struct MatrixUpdateJob : IJobParallelFor {

[ReadOnly] public NativeArray<float3> positionData;
[ReadOnly] public NativeArray<quaternion> rotationData;
[ReadOnly] public NativeArray<int> startIndexes;
[ReadOnly] public int minIndex;
[NativeDisableParallelForRestriction] [ReadOnly] public NativeArray<float3> renderOffsetData;
[NativeDisableParallelForRestriction] [ReadOnly] public NativeArray<float3> scaleData;
[ReadOnly] public NativeArray<int> lodData;
[ReadOnly] public int LODLevelCount;

[NativeDisableParallelForRestriction] [WriteOnly] public NativeArray<float4x4> outputArray;

private int lodLevel;
private int flatIndex;

public void Execute(int index) {

    lodLevel = lodData[index];
    for (int i = 0; i < LODLevelCount; i++) {

        flatIndex = startIndexes[i] + minIndex + index;

        if (lodLevel == i) {
            outputArray[flatIndex] = float4x4.TRS(positionData[index] + renderOffsetData[flatIndex], rotationData[index], scaleData[flatIndex]);

        } else {
            outputArray[flatIndex] = float4x4.zero;
        }
    }
}

}```

flatNativeArrayConfig holds values and some method to access the flat array

[velvet panther]

Can you make the renderOffsetData and scaleData into float4's?

[vital sandal]

well I could but the last value would be wasted

[velvet panther]

Doesn't matter if you waste a float inside the float4, it'll be MUCH faster.

[vital sandal]

oh ok

[velvet panther]

This is coming back to the alignment. and SIMD and aliasing. 4's and 3's don't play well together.

[vital sandal]

only issue is when computing the float4x4 matrix I have to input float3

unless it has an overload

[velvet panther]

There's column maths for float4x4's, is that what you need/doing?

[vital sandal]

nope... I think I tried it but it says there is no implicit conversion but then the doc says there is... or maybe I dont understand what that means

[velvet panther]

Have you watched the video on optimising for Android? It covers this somewhat.

[vital sandal]

nop havent seen it

[velvet panther]

I think it's titled "Burst for Qualcomm" hang on... I'll find it...

[vital sandal]

No im just building an array of TRS matrix (transform Rotation scale) to send to the GPU

its not huge math

but I gotta handle a lot of units

[velvet panther]

First one to watch, this slow, but gets to some very good points; https://youtu.be/BpwvXkoFcp8

This video will teach you exactly what aliasinng is, and how to get float4's absolutely flying through.

[vital sandal]

there will be more complex stuff later, I also think its possible to drop the position, rotation and scale array and just use the float4x4 to get and store those values directly

[velvet panther]

And it might be that you need to rip out the parts of your float4x4's into arrays of float4's to do the work, and then rebuild them.

[vital sandal]

yep I thought about that

Ill have to try it on at least one of my jobs to see the difference

[velvet panther]

Here's the android one, again, very slow, but the meat in the middle is worth it: https://youtu.be/WnJV6J-taIM

[vital sandal]

nice vids, those are exactly what I needed

[velvet panther]

What they're talking about is not unique to Android, it's all CPUs.

Watch the intrinsics video first. It's better, I think, and might really make a few of these things click, as it's aimed at OOP programmers learning to think in terms of SIMD.

[vital sandal]

Hey thanks alot for taking the time to explain all those things 🙂

Im going to watch the videos asap but its 2am now time to wrap up 😛

[velvet panther]

Cheers, good luck!

[vital sandal]

You to!

[prime aurora]

Hey, is anyone up to answer some questions? I'm taking a quick look at ECS and I wanted to know more about how seperate from the old workflow it is exactly.

[left oak]

I believe that Unity's intent is for developers to be able to build their scenes with GameObjects, if they would like to. These GameObjects can then be converted to entities either at runtime, or ahead of time through use of subscenes. However, ECS currently only supports a small subset of classical Unity features, so we're quite far off from effectively utilizing their ideal workflow, and more typically have to home-roll some kind of hybrid workflow. An alternative which is totally separate from the old workflow is to develop what is called "pure" ECS, which is to forgo the intermediary GameObject in favor of developing directly with entities. No solution is currently ready out-of-the-box, and any workflow you choose will require significant individual deviation to achieve an efficient process, at this time.

[warped trail]

i think if you want to make a game with a lot of objects and you are willing to write tools and almost everything else(animation, resource management, navigation, audio, rendering) ecs is your choice, but if you want to make some regular game, just stick with regular unity and look at jobs and burst

[prime aurora]

well, I'm doing almost everything except the bare minimum (transform and renderers) in custom code already, so I feel like it's a good wedding for my workflow

Specifically what made me consider it is that I started writing a batched renderer with DrawMeshInstanced, and I realized ECS is kind of the design I'm trending towards

umh

I'm using MonoBehaviours to interact with the scene, though, to store information like transforms and hook into update. I have a custom player loop thing setup as well but I just find the gameobject/monobehaviour workflow to be more fun for editing

Would it be possible using ECS to write a batch processor for meshes that take some information from monobehaviours at the end of the update loop? Like, transform and frame information for a sprite, say

I saw that a lot of the interchange classes between monobehaviour and entities have been deprecated and it confused me as to how insulated the data is supposed to be

[warped trail]

thats the workflow unity is aiming, game object are just for authoring data

[prime aurora]

I realize having to sync with the main thread every update kind of defeats some of the performance advantages, but I'm not really limited by performance and I dont expect to have more than 50 of these instanced renderers onscreen. I just feel like it would be more straightforward in design as opposed to instantiating dozens of meshrenderers and filters and so on

Like

My actor code can be represented as a mesh or a sprite, depending on the animation data. When switching between the two modes I have to DestroyImmediate the meshrenderer and instantiate a spriterenderer... It's messy as fuck, lol.

Can I just like

Make a singleton in c#, load the instance inside the component system and then fetch values for my actors like transformation matrices and animation data?

I saw an example batched sprite renderer on github that used SharedComponentData to pass transform information to an entity but it appears that the related function calls don't exist anymore

[left oak]

ECS has a pretty decent Transform system, so you could minimize intercommunication between MonoBehaviors and SystemBases

[prime aurora]

How does communication between monobehaviours and component systems work exactly?

Most of the stuff I've found is out of date and deprecated

[left oak]

If you're already using instance rendering, you could just combine that with their mesh api

[warped trail]

communication between monobehaviours and component systems is mainthread only

[prime aurora]

Does that mean the scheduling callback of a job?

[left oak]

like druid said, ideally, gameobjects only exist during editing, and are converted to entities afterwards

[warped trail]

in my humble opinion, for 50 instances it is not worth it to use ecs

unless you have some heacy simulations or something

[left oak]

I agree with this

[warped trail]

heavy*

[prime aurora]

yeah, it's not a performance thing

I just like the model better

it makes more sense since I'm already not really using gameobjects unless strictly necessary

[warped trail]

comunication between ECS and gameobjects is a mess right now

[prime aurora]

hm

[left oak]

here's a link to some docs on jobified mesh creation: https://docs.google.com/document/d/1QC7NV7JQcvibeelORJvsaTReTyszllOlxdfEsaVL2oA/edit#heading=h.vyksohcynwk5

[prime aurora]

I'll read that

[left oak]

maybe see if you can use that

[prime aurora]

Like

[left oak]

between that, and the transforms system, and the instance rendering, you should be able to avoid having any gameobjects to communicate with

[prime aurora]

yes, but I'd like to keep the rest of my project's architecture the same for now

So what you're saying is all in or all out

either upgrade everything into entities or just don't use it at all

[left oak]

you can convert and inject, which basically creates entities of an object without deleting the object

there's some components to hook up to keep the pairs in synce

[warped trail]

but what the point? i think the overhead of ECS will be bigger than performance boost

[left oak]

but it's not really that bad

[prime aurora]

So the renderers associated with my objects wouldn't have to be instantiated in the world

[left oak]

not if they're instanced

[prime aurora]

Objects could just subscribe to the rendering system when required and update their data asynchronously

[left oak]

you could do something like that

if performance is already not an issue, the overhead shouldn't be significant

[prime aurora]

yeah I dont really need ECS for that but it seems like it's the kind of stuff ECS is meant for

[left oak]

you could certainly jobify the mesh creation

jobs and burst are at least worth investigating

[warped trail]

just look into jobs and burst

[left oak]

^-^

[prime aurora]

Is this a realistic idea: basically, keep everything in monobehaviours (physics, game logic, etc) and then have an entity representation that handles the actual rendering and polls data from the monobehaviours such as transforms and sprite indices?

[warped trail]

just try it😁

[prime aurora]

Well, I read up for about an hour or two but most posts seemed to discourage hybridization like that, and the ones where there was reference on how to do it used code that's been deprecated with patch notes leading to github documentation pages that 404...

Like

[left oak]

it seems like all you're really trying to do is pack the data needed for the instance renderer, and you can do that without entities, or objects, for that matter

[warped trail]

just download ecssample from github

[prime aurora]

My rendering is already not really synchronous since I do physics in fixedupdate and interpolate the results in update

This?

https://github.com/Unity-Technologies/EntityComponentSystemSamples

[warped trail]

yeah

[prime aurora]

Last commit was in feb... is it up to date <.<?

[warped trail]

yeah🤣

[prime aurora]

I'll take a look at it tomorrow

thanks guys

I really feel like this is a good direction for Unity but it's pretty esoteric right now since they keep changing the api

I dont even think the architecture is that hard to grasp, but the constant changes are pretty difficult to understand just getting into it now

[warped trail]

i dont think there are a lot of changes in api

[prime aurora]

https://github.com/paullj/unity-ecs-instanced-sprite-renderer/

Pretty much all the central bits of this have been deprecated as of today

and like

It seems like they've been deprecated due to architectural changes too, so there's no way to replicate the behaviour strictly speaking

ComponentDataArray doesn't exist anymore, ComponentGroups don't seem to exist anymore

SharedComponentDataWrapper also doesn't exist anymore

[warped trail]

they just renamed them, basic idea is the same

get your data from gameobjects in linear array, feed it to ecs or jobs, make your computation, copy data from array back to gameobjects

with ecs you can get rid of copy steps

[left oak]

This is old (there's an even older thread that precedes it, but I won't bother linking it) but it might give you some ideas about instance rendering sprites: https://forum.unity.com/threads/200k-dynamic-animated-sprites-at-80fps.695809/

[warped trail]

burst is awesome thing, but there is big overhead of scheduling bursted jobs

if you have to multiply 50 matrices, i don't think it is really worth it

just benchmark it and make your own conclusions

[prime aurora]

I think ultimately it would be worth it not because of ECS but just because reorganizing things to use material property blocks would let me get on that 1 drawcall per sprite sheet

I think like half of the rendering time for my project right now is drawing sprites and other dynamic objects with transparencies :-T

I have destructible fences and things like that that use a ton of renderers

[velvet panther]

Anyone know the absolute fastest way to initialise a new NativeArray<float> to all 1's?

[velvet panther]

Material Property Blocks are AMAZING, but definitely don't need you to be using ECS. They're fully beneficial without it. Maybe more so, as I'm not sure that you wouldn't have to make behaviour (custom) to force ECS to recognise your efforts to use Material Property Blocks.

Check out AnimationLegacy Components, for animating materials. They automatically create Material Property Blocks, and have some kind of low level "access and optimisations" that aren't available to other methods of animating material properties. Caveat... there are some properties they don't work with. Not many, but a couple of them.

If you're not doing 10k plus things at a time, don't bother with ECS. And even then, you're probably better off just bursting and jobbing rather than trying to wrangle everything into and out of game objects and ECS. If you want to do 100k and up, then I think ECS might be worth the pain of having to re-interpret most everything into Unity's VERY VERBOSE ways of doing ECS... which keep changing, subtly.

[prime aurora]

That's really interesting about legacy animations

I've been suspecting that my animation code suffers from not having access to the stuff under the hood but I haven't noticed any differences in performance so far

thanks man

this is why I asked in here. I wasn't sure if it made sense or was worth it

[safe lintel]

"which keep changing, subtly." ha if only

[viral sonnet]

Hey, is debugging working for you? Mine completely broken in 2020.3.18f1 with VSCode and Visual Studio 2019. I can get breakpoints but I can't step forward. It just goes to the next frame

[safe lintel]

i find ill have to restart the editor at times for debugging to work

[viral sonnet]

let's see, i'll try 🙂

restart->enable debugging-> unity crash ... man I don't know why I even bother

I can get more breakpoints but it jumps around erratically. Doesn't seem right. Pretty frustrating. I took some months off because Unity was starting to get on my nerves and now even debugging is broken. lol

[safe lintel]

does it do the same in an empty project?

[viral sonnet]

nah, the project for my work is doing fine. debugging still has its quirks but it's more or less functional. it's just for my dots side project where it's like that.

turned off burst but it's the same. systembase jobs are all withoutburst tag also

it's like the code file is mismatching. it skips so many lines. args

well, at least i found my mistake without debugging. it's like good old PHP time, when in doubt Debug.Log

[safe lintel]

heh thats how I debug most of the time, I find the actual debug with ide kind of a janky experience

[stone osprey]

Quick question... is it possible to use some dot packages outside of unity ? Im talking about the collections for example... like unsafelist or nativelist and fixedstring ^^

[viral sonnet]

afaik no because most are calls to UnityEngine.dll and low level calls to the engine itself. Worth a try still if you copy all the needed dlls

[stone osprey]

What a pitty :/ But thanks.

[viral sonnet]

do you need them for a standalone c# project? which .net version are you using? i don't think you'll ever need them in .net core

it's fast and optimised unlike unity mono

[stone osprey]

Exactly... my plan was to use a c# server which communicates with the unity client.

I wanted to use exact the same structs on my server and on my client... but the problem is that due to burst my client components often contains FixedString and similar stuff. So i thought it would be great to have them on the server aswell

[viral sonnet]

when you send them, you have some serialization in between, right? fixedstring can be sent as byte array, it's compatible with normal c# strings then. i have a .net core server where this is working

[stone osprey]

Well that sounds great ! Yes of course... each struct has a ISerializable interface of my server api which is used to write the primitives ^^
So i guess i just need to change the deserialisation part of the client structs to make it use the collection package, right ?

[viral sonnet]

that's the gist, yes

hm, referencing a subscene gameobject in an authoring component entity was working in the past. for some reason it's not anymore, keeps getting reset to None after exiting playmode.

[gilded glacier]

let me ask again ...

do you guys use some sort of external documentation that helps to design and keep track of your current ecs code ?

like, when it comes to archetypes and queries, it is surely better to have these noted somewhere, especially with the growing project, because there may occur some misbehaviour when you forget to exclude a component form a query, or add one into the archetype for system to process it

[left oak]

Queries belong to their individual systems. I don't personally see a reason to document my queries outside of each given system. Additionally, the UpdateBefore/UpdateAfter attributes at the head of each system are very concise reminders of the potential conflicts between systems that operate on similar queries. To me, the core activity of managing ECS involves considering the flow of each frame in terms of sync points. Sometimes, a flowchart for this specific purpose is useful, but I find the systems to be fairly self-documenting, considering how verbose the boilerplate tends to be. But that's just my personal preference.

[remote crater]

I'm gonna launch my tech demo here as soon as I get "total entities in game","FPS count" and compile standalone.

It isn't a full moba, but still neet

I think I could query all entities with a translation aka position... that should do her. To query all entities

[forest quest]

Use internal fields and public static utility functions to interface with components from different namespaces to effectively create APIs and avoid this kind of guess work.

I like to have a single static utility class per namespace for any necessary APIs that involve one or more components from within that namespace

[remote crater]

I got total entities, FPS, now I just need standalone. This could be fun. Any tips on making a DOTS standalone. What do I need to add to my build?

[timber ginkgo]

can memory leaks result from quitting an application before disposing of native collections?

wondering what i might run afoul of by calling Application.Quit() in various ways

afaik, quitting the game entirely should properly release all used memory right?

[mortal gyro]

@timber ginkgo yep

[timber ginkgo]

Sweet, thanks!

[mental bison]

is there a way to see colliders of entities?

[zenith wyvern]

If you have the physics package installed you need to add the physics debug authoring script to a gameobject in your scene

And convert it to an entity probably, don't remember

[pulsar jay]

I have the problem that collisionWorld.CalculateDistance hits mesh colliders multiple times. Anybody knows why this might happen? Box colliders will only be hit once

[pulsar jay]

Collision queries for mesh colliders seem to return 1 hit per triangle. Is this intended? And can it be changed?

[remote crater]

I haven't been able to destroy entities safely, but I did some cool stuff I want to show and go live with. I need help making a DOTS compile. I already compile to windows standalone, but entities don't render. I think it is very simple to fix for someone, like telling you how to set the clock in your new Toyota: https://forum.unity.com/threads/i-have-wanted-to-compile-a-dots-ecs-standalone-for-about-6-months-i-need-it-now-to-release-any-help.1178299/

[pulsar jay]

It should build with build config without any special setup. Which Unity and hybrid renderer version do you use?

[remote crater]

.17 entity

I need to use windows build platform tho

cuz it does not build naturally since like 10 months ago

Dif question: Is it possible to have an incrementing variable in a .foreach?

I'm not sure it could be... since they're all running parallel

Is there #lock and #unlock for these things?

I would like to #lock the parallel thread grab a variable incrementor, use it, and then increment it

I've done parallel thread coding before, this stuff is easy.

But at the same time, it isn't widely known... So did unity guys put lock/unlocks in, or is there a way to increment a variable?

Actually I can work around on this one...

[pulsar jay]

maybe try using the entitySortIndex?

[remote crater]

This is a variable unrelated to entities being processed, but book keeping stuff

I'm gonna read the data from an entity command buffer

Every time I add one, I'll add a nonsense item

then I can count em after 😉

[pulsar jay]

then it maybe easier to use a nativearray and pass it into the query?! it could even just have 1 int wich you can count in

[remote crater]

I need to write out tho

.foreach does not let you write

[pulsar jay]

yeah I see. didnt think about that

[remote crater]

It makes 100% sense for multithreaded stuff

for i++ inside a parallel job would mean one might be mid read

while another mid write

But if they have #lock/#unlock for this, it'd be perfectly fine

This coding is advanced coding for advanced programmers, we know what lock/unlock is and how it can de-efficify code

so I hope the UNity devs aren't like,"We need to simplify this!"

Instead they should be thinking,"Lets allow all the power of multithreaded to crazy arcane minded individuals. Lets complexify this! But write good documentation."

I am concerned their software development philosophy might not be in line with logic, but I'm just happy they're trying. I can make do with this.

[pulsar jay]

I guess its easier to know in threads you complety setup and run yourself. In this case you need to know the unity system scheduling inside out to know how a lock might effect it

[remote crater]

That completely defeats the purpose of .foreach then

Ok, anyway, I've gone off the trail of hack something to gether that works

and waxed poetic towards code that looks beautiful.

I often make these mistakes, thinking professionals are also masters at their craft. I need to lower my expectations for APIS, even multibillion multinational corps.

I love DOTS/ECS, don't get me wrong, but I just have problems with everyone's api, they always fall short.

[pulsar jay]

I guess you just cant make the "perfect" API

I am still struggeling to find out why a distance query with a mesh collider returns a hit per triangle. this is really unexpected 😬

[remote crater]

Well the perfect API is simple.

You give the user the ability to do anything he could do apart from the API, and then add specifics.

But I'm more in favor of users being able to wreck and go fast and more places than always be on road.

Ok, enough philosophy!

I gotta get a build out, lolz

[pulsar jay]

Might be the equivalent of not using foreach but chunk jobs in this case? This way you can control the "loop" and your counter var yourself?!

[remote crater]

Like I said, I'm fine, have a workaround

I was just saying, if we're to be doing multithreaded code, maybe at first, give us all the basic tools/commands multithreaded veterans are used to.

Ain't no big, no complaint. Just philosophy.

Can you get the length of an EntityCommandBuffer.ParallelWriter ?

Like how many commands you stacked in it?

I'm thinking no.

For some reason I was giving entities indexes

And if I made many of em, had to increment

Now I figure I don't need that

[karmic basin]

Jim I would double check I've correctly followed all setup steps here https://docs.unity3d.com/Packages/com.unity.rendering.hybrid@0.11/manual/creating-a-new-hybrid-renderer-project.html

If that brings nothing I would try to compile successfuly a new project with a simple cube with all package versions you're using, making sure this works at least

• I would remove most of the build components you don't use. Only 3 or 4 of the default ones can be enough to make a simple build

[remote crater]

My project runs fine.

I can not make a standalone. Details here: https://forum.unity.com/threads/i-have-wanted-to-compile-a-dots-ecs-standalone-for-about-6-months-i-need-it-now-to-release-any-help.1178299/

I'm literally not putting anything DOTS related components in my build configuration file. I think that is the problemo.

But when I tried to put DOTS components in my build... It wanted links and stuff to stuff, and I had no clue what that was about: https://crystalfighter.com/pix/build2.png

[late loom]

I have a certain amount of Sprites that I want to use in different parts of my game. Would it be better memory-wise to put all of them in a List<Sprite> and, when I save them on different data structures (such as a list that contains info about an object, including its sprite), only save their index (int) or just save the Sprite straight up?

[velvet panther]

Is there a discord for discussing optimisations of Burst, Jobs, Vectorisations, SIMD/AVX considerations etc?

[shell berry]

There's a separate forum for Burst.

Not in Discord, though.

[shell berry]

There's not much difference because the Sprite that you're getting are references (could also be pointers if you like that mental model). In fact, there would be extra overhead when using indices because you'll have to used them on the list whenever you want a Sprite reference.

[remote crater]

In my standalone: Entities are in game, just not rendering. I use HDRP if that is any consideration. Collisions happen. Rendering does not.

Again, can someone please check: https://rumble.com/vncywp-starfighter-general-standalone-problems-no-rendering-of-entities.html

It is a very short video, just under 2 minutes.

[karmic basin]

Uhm... okay we see more with the video. Could you :

1. Paste the error here ? It's about the scene not added to build settings though you have it in legacy build settings, right ? If so try number 2 :
2. Add your scene (the one in capital letters) explicitly to the build list in the build config asset (button underlined in blue in the screenshot), right now you only build the current scene for DOTS
3. I would only keep the 3 first build components (red in the screenshot) unless you have good reasons to keep the others (honestly I'm not even sure what they all do), eventually keeping the output directory override one.

I bet on the second point being the solution, keep us updated

[timber ginkgo]

can I reallocate a persistent native collection (like a nativeHashMap) without disposing of it first?

not sure if i should just set an upper maximum and not bother resizing it though...

[remote crater]

Can someone look at this video? My entities in Standalone are not visible: https://www.youtube.com/watch?v=7N6y56Ij2Es

What could make entities not render in standalone mode?

[remote crater]

Since we have to rewrite all a component to change one value in places... Is there a .clone() function on all iComponentData?

[forest quest]

You're confusing classes and structs.

[half jay]

Is it possible to use URP and Hybrid Render V2 with DOTS on IOS/Android?

[karmic basin]

Did you see my message just a little bit above ? #archived-dots message

[remote crater]

It still gives same error: "Scene 'NEWMOBATOENTITY2NON' couldn't be loaded because it has not been added to the build settings or the AssetBundle has not been loaded.
To add a scene to the build settings use the menu File->Build Settings..." https://pastebin.com/RLUzCF0T is the full error log.

Here's my build pic:

I also tried checking auto load.

[remote crater]

Thanks for trying Mr.K. I'm working on this brick wall and my brick wall of not being able to destroy entities. If I pass them both I'll have a MMORPG & MOBA out in about 1 day.

[half jay]

Can i use property of system instead of field in burstable systems?

[amber flicker]

sorry - realise that was unhelpful but in a meeting - I think the Unity exp right now is going to be very rough. I don't think the terrain shaders work with the hybrid renderer for example. There is an asset on the store (Orbis) that looks kinda interesting but I've no idea what it's like or if it would help you - what it does (if anything) about the shader issues etc.

also have you seen this thread? https://forum.unity.com/threads/using-unity-terrain-with-dots-workflow.755105/

[karmic basin]

Uhm alright can you show how you load the scene at runtime ? In here : StationBoard:launchMoba () (at C:/UnityProjectfiles/StarfighterGeneralOnSteam/Assets/Scripts/Misc Scripts/StationBoard.cs:11615)

And Also what happens in this method : Controls:FixedUpdate () (at C:/UnityProjectfiles/StarfighterGeneralOnSteam/Assets/Scripts/Player Scripts/Controls.cs:1497)

Let's see if you try to move the player before the scene is loaded

[digital kestrel]

anybody have an animation to texture baking tool that bakes the bones position into the texture

i found a tool for baking vertex positions but that's not very efficient

[remote crater]

SceneManager.LoadScene("NEWMOBATOENTITY2NON", LoadSceneMode.Additive);

More info: If I deliberately do not allow the scene to be loaded in the editor, the entities still render.

[remote crater]

More info: If I add gameObjects to scene they aren't seen either. I'm looking into group options now.

This question is probably identified as non DOTS related at this point, more with Addressables package.

[whole gyro]

Do you need multiple bone weights/influences per vertex? If not, you could take a look at this: https://storyprogramming.com/2019/09/18/shader-graph-rigid-body-animation-using-vertex-animation-textures/

It bakes object positions instead of vertex positions, so this requires different game objects for each bone.

[digital kestrel]

I believe I do or will need multiple bone weights. I know in the Unity Austin Tech demo, they baked everything into a texture and read from that. I think im going to do the same lol @whole gyro

im looking through the github project to do that, but I was wondering if anyone already ported it to a tool yet

[whole gyro]

Yeah, I briefly looked for the same thing and didn't find it anywhere. Let me know if you do! For now, I switched to using the dots animation package since I realized I want an entity to track the bone positions for my use case. Hard to do something like that with any kind of animation baking that relies on a shader for playback since those positions are calculated on the gpu. Maybe something with compute buffers would work though.

[digital kestrel]

yeah i wanted to spawn more entities by lowering cpu overhead from things like culling system and calculating bone positions and instead pass the workload onto the gpu using Graphics.DrawMeshInstancedIndirect

@whole gyro if you want references, go to Unity Austin demo github and look at TextureAnimatorSystem.cs, Standard.shader, and KeyFrameTextureBaker.cs

[whole gyro]

Thanks!

[shell berry]

Oh, it uses DOTS?

[mental bison]

i know there's a way to make an entity's collider a trigger in the inspector with physics shape script. But how would one do that at runtime?

[half jay]

i have this method which throw exception about that queries doesn't exists that there are 0.

but i have this entity with single component in the world

What can be wrong or im missing something ?

[karmic basin]

The collision response defined in the PhysicsShape component is set into the collider Material during conversion.

So you're looking to change Material.CollisionResponse property, which one of the values can be CollisionResponsePolicy.RaiseTriggerEvents

[mental bison]

aha i see

[karmic basin]

also check my previous message about it, someone asked a few months ago #archived-dots message
I didnt play with physics for a long time but it should work

if not correct me I'm curious 🙂

[karmic basin]

i have this method which throw exception about that queries doesn't exists that there are 0
What ? Say again ? 😅 didnt understand the question. Or paste directly the error

[half jay]

its throwing me this error

GetSingletonEntity() requires that exactly one entity exists that matches this query, but there are 0

[karmic basin]

When is it called ? Are you sure it exists when you query for it ? Sorry dumb question, just doing the rubber duck here

When i say when: in which system is it created (or is it at authoring time), and in which system the get is done ?

[half jay]

IgnoreResetApp() called from Monobehaviour and system which called GetSingletonEntity showing on 2nd screen and Query for this CurrentStateSingletonTag exists

[karmic basin]

ok. so _system is MainApplicationSystem, retrieved with a GetOrCreateSystem<>() in the MB

the 2nd screenshot is stuck on the exact frame of a breakpoint ?

trying to figure out if IgnoreResetApp() is called too soon but other than that I don't see why

[half jay]

yes. And this entity with this CurrentStateSingletonTag never destroy. If it will happen ill get tons of another errors and it will obvious

[karmic basin]

I got no other idea, sorry. I'll let someone else suggest things

[half jay]

np ty anyway

[gusty comet]

Hey everyone, i am trying to convert my existing project into hybrid ecs. I need to use it on my npc's for pathfinding and etc. I have about around 300 of them at once in one scene so it lags up the game a lot. Thats why i need ecs. But i coudn't find any recent tutorials on this topic. So i am open to suggestions

(keep in mind that i just started using ecs)

[safe lintel]

No official native dots navigation so might not be worth a switch @gusty comet

[safe lintel]

also theres the whole dots being on a public hiatus 😩

[shell berry]

We rolled up our own. It's grid based, though. If you know how to implement A* for your game, you can definitely turn it to DOTS. The following might help:

https://coffeebraingames.wordpress.com/2019/04/14/how-fast-could-burst-compiled-a-be/

https://coffeebraingames.wordpress.com/2019/05/19/from-160-to-300-a-requests-per-frame/

https://coffeebraingames.wordpress.com/2019/06/21/replacing-our-oop-pathfinding-to-pure-ecs/

[robust scaffold]

Just a minor note, I've done some testing on systems and there is about a 10% performance loss between Ijobchunk (0.38ms) to Ijobentitybatch (0.41ms). Just a personal observation using the profiler, nothing scientific.

vs

Using IJobEntityBatch with a chunk subdivision of 2. It definitely utilizes all threads, well more often, but I've already shattered my chunk organization with 2 different shared components so IJobChunk is actually fairly well distributed.

Also, if you want to know what new API / changes new DOTS is going to bring, keep a close eye on this github repo and the various forks. They're being coded in DOTS 0.20 (public is 0.17) https://github.com/Unity-Technologies/DOTS-training-samples/tree/2021-09-europe-grp2-parallel-constraints-hacks

[pliant pike]

300 shouldn't lag the unity nav system, I have 1000 using it with high fps

[robust scaffold]

My current observations of Entities 0.20:

1. Added GetSingleton<>() for managed component singletons (Class IComponentData) in SystemBase. Dont see any references to "SetSingleton" though.
2. DOTS is still lambda based except for when chunk components are needed, then it's IJobEntityBatch. Real shame, wish they can do chunk components in lambdas.
3. Unity really loves the use of ECBs. I personally have a bad taste of them since back when they were introduced. Maybe I should start using them.
4. Probably something specific to DOTS editor but Monobehavior -> UnityMonoBehaviour?

[gusty comet]

but i am developing the game for mobile

[safe lintel]

@robust scaffold using UnityMonoBehaviour = UnityEngine.MonoBehaviour;

only at 0.20.x, was hoping to be further along after all this time

[robust scaffold]

Ah, didnt see that. You're right. Also there doesnt seem to be that many improvements / major changes at the surface code level. None of them are using struct ISystemBases so I'm thinking bursted everything is on hold. Shame, I was looking forward to that bright future.

[gusty comet]

How are things on the dots front. Havent checked in for a while.

[digital kestrel]

no updates for more than a year lol

[coarse turtle]

what front 🙃

[gusty comet]

so typical unity then, hype up a feature, ship it half finished and then abandon it.

[safe lintel]

Havent checked in for a while. neither have unity HEYO!

[gusty comet]

its just a shame because ECS has huge potential for unity. I love ECS as a design paradigm and a unity where that is front and center would be a godsend for me.

[safe lintel]

yeah im feeling quite sour on this now, after kind of being really positive on it despite its flaws.

[gusty comet]

Well I am not as sour since I have never used it since they had no built in 2d support out of the gate and I exclusively make 2d games.

[molten flame]

Has anyone had success integrating win32 apis into their game?
I just want to move my main player window but I cannot get it working, it just crashes.

I know this is probably unrelated to DOTS but I was just wondering if there was something in the build script process that was different that I need to do

[half jay]

Where can i find any info or docs about this?
https://docs.unity3d.com/Packages/com.unity.platforms.android@0.10/manual/index.html

Is this a new build pipeline? Is it always necessary now? Our project still on 2020.1 and we use DOTS but build it with default way through player setting

[vagrant lotus]

is there a way to **not **copy the **scale **when using CopyTransformFromGameObject?

second question

does **colliders **get affected by scale?

[karmic basin]

Yes you need a different build pipeline for DOTS. Especially if you have subscenes, as the legacy Build menu don't know what that is. You can read a little about it here https://docs.unity3d.com/Packages/com.unity.entities@0.17/manual/install_setup.html#standalone-builds

Make sure you have com.unity.platforms as well as com.unity.platforms.android, but now that I write it, I'm sure it should come as a dependency

[half jay]

tyvm, hopefully we are not using subscenes we have convertation couple gameobjects

[karmic basin]

Looking at source, the scale is always forced to 1

[vagrant lotus]

thx
i know its bad is most case
but this is exactly what i wanted in this case

[karmic basin]

If I remember well, no. Colliders are baked into blob assets (so immutable data) at conversion time. You would need to create a new collider with a new scale I guess. Might wanna look at the samples, if I remember well they have a scene where they mess with colliders shapes, maybe they demonstrate also scale ?

[vagrant lotus]

yeah im doing that

i was worried if the scale at applied twice (by me and unity)

curious question:
is there stuff in 2022 alpha that is** dots related **(i.e. editor tools etc.)?

[karmic basin]

Didn't try yet but versions above 2020.3 are not supposed to be supported

[vagrant lotus]

ye the memo says compat will only return at earliest this year's end

[karmic basin]

yeah

[vagrant lotus]

finger crossed

[karmic basin]

Some people reported here that some of the packages broke

or at least one but I don't remember which

[vagrant lotus]

probably just entities 0.17 not playing nice with collection 0.17

😉

[half jay]

I have crash on my app using "com.unity.platforms.android" any ideas what should i check?

and my config

and are this setting depends on Player Settings or override them?

[rancid geode]

if I remember correclty, dots runtime is Tiny, which doesn't support AR for now

[pliant pike]

basic question but Nativearrays when gotten will always stay in the same order wont they?

Like as long as I don't change the size of the array, or remove or add anything then I can use the elements of the array as unique IDS right?

[dense crypt]

As long as something else didn't modify the array , yeah

[pliant pike]

figured, thanks

[vagrant lotus]

since** entitymanager.iscreated **is now gone, how do i check if the manger is disposed or not?

[molten flame]

Has anyone had success automating a build for DOTS? how do you do it?

[molten flame]

You could check the world maybe?
Why do you need to do this? or if I can rephrase, what are you trying to acomplish?

[vagrant lotus]

accessing ecs from monobehaviour

solved by checking if is application close time

the problem is that onDisable get called when closing app which by then entity manager is diisposed

[remote crater]

Easy Question for someone: In a .parallel burst .forEach, how do I set any sort of data to be read outside the loop? Is there an IComponentData singleton type of thing shared among all entities that have it?

[pliant pike]

just set the component data that is on the entity?

[remote crater]

Is it possible to have static component data that all entities can read from? Like a Singletonish variable? I need this data to be semi global and read other places.

Right now it is just a debug flag boolean, true or false.

[pliant pike]

you can't set a singleton or a single nativearray, because its 4, 8, 16 or who knows how many separate threads/jobs running at once

[remote crater]

I want to destroy once, and never again. Is it possible to read the length of EntityCommandBuffers? Any sort of data exiting the system would be good. I mean worse comes to worse, I could query all entities, and find the data.

So there is no way for data to exit a .foreach unless tied to each specific entity?

[pliant pike]

trying to do that is a race condition which is undesirable behaviour

[remote crater]

I know

Its worse than race

its memory corruption to set and get a variable at the same time

I am aware of this.

I was just wondering, is there any way for data to exit the .forEach except in each entity?

Normally I just use #lock/#unlock for these things, but it seems like ECS denies the user the ability to use lower level parallel programming? That's probably for the best for most users.

[pliant pike]

you won't always end up with memory corruption, I've done it before accidentally, you just end up with completely wrong values returned

[coarse turtle]

you want a read/write static variable? Probably shared static might be what you're looking for.

[remote crater]

A static inside the Update cannot be accessed in .foreach

[coarse turtle]

https://docs.unity3d.com/Packages/com.unity.burst@1.6/manual/docs/AdvancedUsages.html#shared-static

[remote crater]

ISharedComponentData ?

Thanks itsJustBlank

iSharedComponentData is like a Singleton shared among all entities right?

I think this is what I need, thanks guys, we on right track. God bless, and if I make bank, find me, and I give you some bank.

[coarse turtle]

I didn't state ISharedComponentData, just Shared Static variables

See the burst link

[remote crater]

Is there any syntax examples of this? I tried: public static SharedStatic<bool> destroyedOnce2;
and it won't let me get or put to destroyedOnce2;

[coarse turtle]

Yea the code snippet is in the link?

[remote crater]

Did they make their own class? Like is that shared static thing its own class? Referenced in another class? Its two snippets

Or is that all in one file.

Cuz I am not sure if it is allocating for getters and setters or something?

Eh, I think I'm bothering you guys too much

[coarse turtle]

Yea the 2 snippets

As long as you declare the static readonly variable, so it's only assigned once

you can grab the static variable and mutate it

[remote crater]

So if it is read only. How do I write to it in the .foreach?

[coarse turtle]

the SharedStatic<T> is effectively a container where you can write your data into it

readonly is to ensure that SharedStatic is never assigned more than once

[pliant pike]

how does that work in a parrallel 😕

[remote crater]

So despite it being read only, I can set it once? Per thread execution or just at declaration?

This isn't making sense. Maybe there is miscommunication here.

[coarse turtle]

    // Write to a shared static 
    MutableStaticTest.IntField.Data = 5;
    // Read from a shared static
    var value = 1 + MutableStaticTest.IntField.Data;

The Data field is where you assign your variables

[coarse turtle]

probably need some queueing mechanism tbh, I've only use this in IJob.

[remote crater]

So its read only that you can write to once. Weird. Thanks. I'll try that code.

[coarse turtle]

And I prefer per thread containers for multithreaded executions versus static single variables

[pliant pike]

hmm I didn't know there was something like that, so you could have it so the first job that sets it locks it from all the others 🤔

[remote crater]

If this works, it will be hyper cool!

[remote crater]

It works to the effect that I would like. So now I know two ways of getting data out of a .foreach sharedstatic and sharedComponentData. It is a good day when you learn something useful in DOTS and just aren't brickwalled. An interesting thing to note: this sharedstatic is not locking/unlocking in nature for some other parallel threads can get the old variable value before it is fully set. So save it for data you want to exit the .foreach and not data you want to use by other entities in the .foreach, but that is to be expected. Thank you itsjustblank

This will sound quite dumb, but is there some way to disable burst on .ScheduleParallel(this.Dependency); in order to get a better debug output?

[remote crater]

This global shared static is helping my debug SOOOOO much. I can test in even other jobs with it, so I'm moving it down my job in binary search fashion with a if(sharedstatic = set) return; This means, I can find out where one line of code in one job messes up another line of code in another job (burst does not give line number info)

[coarse turtle]

use Entities.ForEach((...) => { }).WithoutBurst().ScheduleParallel(...)

[remote crater]

ty

Thats even better!

[coarse turtle]

You could also try wrapping Debug.Log in your own custom wrapper and use FixedString instead

[remote crater]

The old school way of being unable to use a debuger and tracer, is to put lots of print statements, but that comes to a challenge when 100 entities of print statements go on at once.

Low key, I enjoy the challenges of DOTS/ECS if I wasn't trying to crank out a MMORPG yesterday lol

Unity did an astoundingly good job on this.

I can't wait to launch MOBA/MMORPG and give everyone the run down on how to do this easy even with their old video games ported up on a youtube video

Well this is odd... Apparently I am unable to instantiate a EntityC from EntityA when EntityB was instantiated from EntityA and EntityB was destroyed.

[tight blade]

anybody have resources/vid links for good input handling in ECS? I've seen videos that use static keycodes. but everything refers to a better way of processing inputs

[remote crater]

I just use monobehavior on my old game's monobehavior

Then I sync my player entity with it

It's actually the "right way" to do it too

If you're porting an old game

Because if you place input in two different player controllers, you have to update both non stop.

What you want is like a phantom GameObject that mimics the Entity, but takes input and processes to entity.

GameObject that has no collision and does not render

Its complicated to understand, but easy when shown. I want a tutorial out this week or next.

[tight blade]

right, yeah, that all makes sense

[remote crater]

Its Design Pattern proper code to not rewrite twice

One player does not need parallel processing

by definition you don't have 1000 keyboards and players on the pc

As long as the enemies and events n stuff are ECS, boom, you good to go.

[tight blade]

I'm in the weird state of understanding the ECS world (such as it is), and being kind of ignorant of a lot of Unity's GO systems. most notably including build tools, animation systems, and relevant to the current question, the input manager

I have very spotty knowledge without much breadth. I'm a physics, job scheduling, and 3d quaternion math pro at this point

but quiz me on chapter 1 of "how to unity" and I'll fail

[left oak]

The new input system has had direct ECS compatibility on its wishlist for quite some time, but has not yet implemented it, as far as I know

[robust scaffold]

Just putting this here: Loop vectorization is amazing. I have to jump through so much hoops and I have no clue what I'm doing with these dangling pointers but Loop.ExpectVectorized(); returns true and performance goes from 77ms to 4.51ms doing the exact same code.

I need to add a value to every transaction component, which is just a wrapper around a float.

• Non-vectorized, which is just a naïve implementation of addition straight into a reinterpreted NativeArray<float> over about 1,500,000 entities results in a total computation time of 77.31 ms.
• By relocating the addition to a separate function using instead a pointer to the NativeArray containing the components, Burst can successfully vectorize the addition loop and cut down total computation time over the same amount of entities to 4.51 ms. Absolutely amazing.

This is however my first time using pointers, had to turn on Unsafe code in the preferences to implement this, so I have no clue if I'm doing it correctly. I dont think there's any memory leaks... I hope.

[coarse turtle]

no mem leaks there, since you're just accesing the array's internal pointer

[robust scaffold]

Alright, thanks

[tight blade]

wait, so main scene camera can not be managed with ECS, even with a hybrid render, is that correct?

[robust scaffold]

Hybrid renderer just creates a new entity with translation and rotation that is copied to a "hidden" camera gameobject.

Along with, i believe, other camera info. Changes to the "camera" entity is just directly copied to the camera gameobject on the main thread by various systems

[tight blade]

hmm, I was referring to this line in the hybrid renderer docs:

what I'm trying to do is create a simple camera follow scenario by making the camera a child of the player entity

what you're describing seems like it would be perfectly fine to handle that use case, but this line in the docs seems to suggest that it isn't possible at the moment

@robust scaffold

[robust scaffold]

Oh, hrm. Thought that camera info was also a component type. I mean you can code it yourself as well but the transform I know for certain is only a "surface" wrapper for the actual camera which remains a GameObject

There are typically a handful of Cameras. Conversion to entity doesnt make much sense rather than just directly handling a camera gameobject in the main thread

[tight blade]

well, provided the GO Transform is synched with my entity's translate/rotation every frame that's perfectly fine

[robust scaffold]

Yea, that's what it does.

[tight blade]

huh, so am I correct that you're saying there is not a way to create such a thing with the main camera, and that theres not a strong use case, because you could just code the transform updates yourself, @robust scaffold?

if so, then I guess the next question I need to investigate is "how do you expose ECS data for access from a Game Object", right? Im not saying you need to answer that question, I'm just saying that if I understand you right, thats where I need to go next

[robust scaffold]

To get ECS data, access the EntityManager and GetComponent<Data>(Entity). Of course, you need to know the entity first before accessing it. Then it's as simple as GameObject.Find("Name Of GameObject") and then going through the standard mono way for a single entity or IJobParallelForTransform for multiple transforms on GameObjects.

[tight blade]

okay. I think I understand. I'm a GameObject novice, so this will be new knowledge for me

Ive seen some videos where gameobjects can be given an Update function which runs on the main thread, so I think, from what you're saying, I could add a monobehaviour script to the camera which instructed it to set its transform values every frame

[robust scaffold]

Yea. Thats what I do for my camera movement script. The only monobehavior script in my entire project.

[tight blade]

outstanding, thank you. And how do you expose the entity ID? Presumably you don't know the entity ID until after the player has been created, right? Do you put it in a singleton or something?

[robust scaffold]

Either making the camera transform entity a singleton (by attaching a unique tag component [empty Icomponentdata struct]) or just storing the entity as a private property on the monobehavior.

You'll need to hook into the conversion workflow to grab the Entity reference if you're using the automatic conversion method or just expose it publicly for the monobehavior to find if you manually convert it.

[tight blade]

huh, the public variable solution sounds lovely if thats gonna work - definitely gonna try that

[cerulean pulsar]

Burst question:

Do the speed ups from Burst come only from bursting a job that works on many entities? Or, might it be worth it to jobify and burst even a single-entity task, like a character translation-rotation update for a single character that involves some non-trivial math (e.g. decollision procedures)? (There would be job-scheduling overhead by default but maybe that could be avoided via .Run())

Will profile to test for my exact case but I'm interested in others' thoughts

[left oak]

Burst is orthogonal to entities and speeds up most jobs, especially ones that can be vectorized. Just take care that if you're using it in a job that only runs once that you use synchronous compilation so that the job doesn't run while the bursted job is still compiling

@tight blade, just add this (https://docs.unity3d.com/Packages/com.unity.entities@0.17/api/Unity.Transforms.CopyTransformToGameObject.html) to your camera entity and point it at the main camera game object, and the transform will be automatically synced for you

[tight blade]

😮

what? are you for real?

[left oak]

Yeah, this works for any dual entity/gameobject where you need the gameobject to exist but want to handle its transform with ecs

[tight blade]

wait, it doesn't have a slot or anything in the inspector for associating it with the camera object.. how does that bit work?

should I make the camera object a child in the hierarchy or something? (wild guess)

I guess I can read the script.

[left oak]

I dunno to what you're trying to parent this camera, but copytransformtogameobject is to be used during conversion (specifically, convert-and-inject), which is how it establishes the gameobject-to-entity relationship

[tight blade]

ohh, I see, you you place the component on the same hiearchy element that has the Camera GO Components onto it, and use the convert and inject

[left oak]

Yeah, I usually just add it in the Convert method of my authoring script

[tight blade]

ah, I see what you mean. It's just a simple struct that you can add yourself

[left oak]

Yeah, and the gameobject will now have its transform updated through the TransformSystemGroup

Although, I think actual gameobject synchronization might occur during the initialization group, I'm not sure

[tight blade]

damn, I'm getting some kind of errors about the referenced entity not existing, so I might need to read through that CopyTransformToGameObject component's conversion flow or at least go find some examples. Anyway, this is a huuge help, thank you!

[left oak]

No prob. There's lots of forum posts about using this struct. I'm sure someone's tried to use this on a hierarchy. I've only used it on unparented GOs

[tight blade]

Nice. yeah, this gives me great leads

yesss got it!!

thanks, @left oak ! Working like a charm now.

[robust scaffold]

In one hour of coding, I spend 55 minutes staring at my computer thinking of how to structure my entities to allow for vectorization and 5 minutes of actually coding it.

[robust scaffold]

And then I sit here for another 30 minutes realizing my previous structure doesnt work for the next step of the system chain...

[karmic basin]

Reminds me of my junior years when I had hard times to figure out how should I build a framework architecture from scratch. Stuck in my head :)

[pliant pike]

Same, but I probably spend a few days trying to figure the structure out

It's at least quicker than OOP though, I got completely stick in that

It's much better with dots implicit structure and goals

[gusty comet]

just delegate the problem to the expert in that field

the future you

[karmic depot]

hey. Can someone explain why in Netcode, the Time.ElapsedTime called in ClientPresentationSystemGroup diverges over time compared to Time.ElapsedTime GhostInputSystemGroup?

[robust scaffold]

Hrm, is there any way I can loop vectorize the summation of an array?

[amber flicker]

Don't know specifically about netcode but if you're accumulating delta values in systems that have differing rates, you'll certainly get discrepancies due to errors accrued using floats. No idea if that's what you're seeing though.

[remote crater]

in a .foreach, I have: ref Rotation rotation. Can I get LocalToWorld from that without costing one of my precious .foreach parameters?

There might be some way to do: quaternion myLocalRot= rotation(SOMEHOWMODIFIED);

To be specific, I just want LocalToWorld Location. I need my ship's facing to aim a laser out the front.

[robust scaffold]

If you're running up against the foreach default parameter list, consider using IJobEntityBatch struct to define your jobs. Or use custom delegates: https://docs.unity3d.com/Packages/com.unity.entities@0.17/manual/ecs_entities_foreach.html#custom-delegates

You can also get significantly better performance by coding for loop vectorization manually in IJobEntityBatch rather than hoping and praying that Unity does it for you.

[remote crater]

Thank you. I know about that delegates KornFlaks, I don't understand the syntax/filesystem of that. I think I'll learn it later, people said I'm doing something wrong if I have too many anyway.

They said its better to crunch all your data into larger but less components

I was just wondering if it was possible to get LocalToWorld from just a rotation?

[robust scaffold]

I dont believe you can.

[remote crater]

TY much!

[robust scaffold]

Yea, consider splitting up your jobs / ForEach into multiple. Typically you shouldnt be running up against the parameter max for a single action in a system.

[remote crater]

I definitely did!

I took my one line of propulsion that required physics mass

Ans stuffed it in another.

I'm getting the hang of DOTS. And frankly, I like the puzzleness of it.

I feel like Unity did a splendid and super exceptional job on this tech.

Parallel processing was something we always wanted to do since late 90s, but knew it wasn't the time to tackle it until we got hardware bottlenecked and had many core computers

[robust scaffold]

Same. I'm working with about 1.5M entities and making sure every action I'm doing to them all are either vectorized or negligible in computation time is so challenging.

[remote crater]

That's cool. I can't wait til I can do a standalone.

I can do 500k no probs

It reminds me of being the 80s crew of arcade and nes gamers, and watching the game manufacturers try and make games we still found challenging.

Like I'm dorking out over the complexity like an old man around a 10,000 count puzzle table.

hahah

[robust scaffold]

Just spent 3 hours going back and forth between the burst inspector and my code. Literally 20x speed improvement from default chunk iteration to vectorized chunks.

~98ms default foreach loop to ~5ms vectorized for loop. Amazing.

[remote crater]

cool

Dumb question: I have the forward rotational vector of a space ship and when it thrusts, it goes forward. How do I pass that vector on to a laser, LocalToWorld or something?

I had it working before, but I think my backups might be lost to space.

[sleek chasm]

Just finished all the videos in the https://dots-tutorial.moetsi.com/unity-ecs/intro-to-unity-ecs and feel like I have a pretty clear picture of how this is actually supposed to work

Now to actually make something lol

[remote crater]

Ok, so you can aim a laser just on rotate... It's just my only math weakness I know is matrix algebra and Quats, lol. Found out how to rotate a quat 90 degrees: myQuat =myQuat* UnityEngine.Quaternion.Euler(90, 0, 0);

This DOTS Video I made is funny... And frankly... I don't even know why. https://rumble.com/vnkfjt-why-is-this-funny-ecs-dots-burst-unity.html

[green ledge]

I've just heard about the term ecs as well as dots, do you think this is a good resource to learn dots/ecs?

[robust scaffold]

Yea, that tutorial is possibly the most fleshed out and explained full project guide to DOTS + Netcode + DOTS Physics + UITookit. Basically the (very sharp) cutting edge of Unity right now.

So cutting edge that it's broken in 2020.3 due to package bugs.

However it's not the "best" to learn DOTS from. It makes some, personally, questionable choices in component structure that I wouldnt do from my ECS structure but it works well enough.

[robust scaffold]

And DOTS is basically Unity ECS. Term's interchangeable.

[green ledge]

Oh too bad, I like to learn "best practices" from the very beginning

[robust scaffold]

There really isnt any for DOTS at the moment outside personal preference. "Best practice" will be ensuring loop vectorization as much as physically possible but that is difficult to explain if you do not already understand DOTS inherently.

I've been coding in unity ECS for about 2 years now and I'm still wrapping my head around vectorization. It's revolutionizing my view on entity structure.

[green ledge]

All that i understand from dots is that it's composed of job system, entity component system and burst compiler, job system allows the implementation of multi-threading to be easier and ecs is new way of writting code (it differs from monobehaviour)

Currently thats all I know ( correct if iam wrong btw)

[green ledge]

First time I come across that term "vectorization"

[robust scaffold]

Yep. ECS arranges data in a way that really leverages the multithreading that Jobs provides. I'm working with 1.5 million different entities with all different data and I can still run at 300fps+. With data changing every frame.

[robust scaffold]

Vectorization is like "macro" array operations. For example, you can add Foo[1] + Bar[1] = Result[1] and Foo[2] + Bar[2] = Result[2] and so on in different operations. But with Vectorization, you can accomplish these operations simultaneously in a single assembly call.

Kinda like x + y = z can now become (x_1, x_2, x_3) + (y_1, y_2, y_3) = (z_1, z_2, z_3) in a single addition operation. If x + y = z took 10 ms to complete and there were 3 elements per variable to add together (resulting in 30 ms runtime), (x_1, x_2, x_3) + (y_1, y_2, y_3) = (z_1, z_2, z_3) doing all 3 elements at once would result in 3x faster operation (resulting in 10 ms runtime).

If you needed to do that operation every single frame, 30 ms runtime would result in 33 FPS. Vectorized operation would result in 100 FPS. 20ms may not seem like much but that's 67 additional frames per second.

Going from horrible game to actually playable.

[robust scaffold]

Tip of the day: Interlocked can be used on components obtained from GetComponentDataFromEntity if the component itself contains the pointer. This allows for the parallel summation of values from other entities into a different entity without needing to go through a slow NativeMultiHashmap<Entity, float>() accumulator. Reading the Burst documentation is very useful indeed.

[BurstCompile]
private struct IteratePopChunks : IJobEntityBatch
{
    [ReadOnly] public ComponentTypeHandle<CitizenshipChunk> Citizenship;

    [NativeDisableParallelForRestriction] public ComponentDataFromEntity<Census> Census;

    public void Execute(ArchetypeChunk batchInChunk, int batchIndex)
    {
        var countryEntity = batchInChunk.GetChunkComponentData(Citizenship).Country;

        unsafe
        {
            // Enabling interlocked pointer from component data from entity cuts time
            // from 4 ms as single-thread to 0.07 ms multi-thread (0.1 ms total)
            Interlocked.Add(ref Census[countryEntity].Ptr[0], batchInChunk.Count);
        }
    }
}```

[hollow sorrel]

cool trick, using pointers directly to reduce component fetching overhead is interesting
seems like you lose some determinism in the order things are processed (in case other systems want to access that ptr's value at the same time) but I guess that's the tradeoff

// Enabling interlocked pointer from component data from entity cuts time
// from 4 ms as single-thread to 0.07 ms multi-thread (0.1 ms total)
how many threads? what is the total time spent across all threads?
are you comparing using interlocked vs setting the component values? or interlocked single thread vs interlocked multi?

[abstract flume]

ive started getting these errors out of nowhere, a few minutes ago everything was working fine with the same setup

[amber flicker]

I don't understand what this is doing. Would like to learn more. A couple of lines usage after this would help me get it I think?

[abstract flume]

seems to be all fine again, after updating from 10.6.0 to 10.7.0

probably just needed cache updating

[green ledge]

I see now how it can get very complex. (Talking about vectorization)

But what does it have to do with ECS? is it like an imporant aspect of it?

[green ledge]

And also is it ok to learn ecs from a resource that is 2 years old? I just thought maybe there is gonna be some drastic changes.

[coarse turtle]

it might just show you the basics, but I'd recommend comparing that with the official dots samples (see pinned links in this channel) and looking at the source code

[rugged bridge]

If the game already runs okay on most systems, learning DOTS is not needed, right?

[robust scaffold]

how many threads? what is the total time spent across all threads?
I'm running this on 16 job threads. Also a typo, it's ~1 ms total (not 0.1) across all ~0.07 ms per thread (0.07 x 16 = 1.12, it varies).

are you comparing using interlocked vs setting the component values? or interlocked single thread vs interlocked multi?

• Previously, I was accumulating sums using a NativeMultiHashMap that took in destination Entity as key and the batch.count as value, then another job that operated entity-wise and summed the NMHM to a single value.
• Then I removed the NMHM and directly added the batch.count into the component of the destination Entity. However that means I could not do this summation in parallel so it was a singlethreaded job, which took about 4 ms for ~1,500,000 entities fragmented significantly due to shared components.
• I then thought, I can sum these values using interlocked if only I could get a pointer to the value from GetComponentDataFromEntity but the pointer itself is never passed to the caller and I dont want to rip open the Entities package to make it public.
• So instead I now made a pointer on the component itself, malloc a single 4 byte sized array to fit in a single int and a pointer to it that can then be passed to interlocked directly.

The main drawback of this method is the lack of vectorization, at least in all my attempts. Xchg, the assembly code used for Interlocked, does not have a corresponding vectorized assembly command and setting the value of a pointer within a pointer does not seem to vectorize at all (probably due to the fact that while the pointers are aligned, the value they are pointing at may be anywhere in the memory).

[robust scaffold]

There are a lot of entities, organized in chunks based on their "citizenship" to a "country" entity. CitizenshipChunk is a component with a single property, the country entity.

Every tick, each country wants to know how many "citizen" entities live in that country to distribute money (in a later system).

To calculate the quantity of citizens per country, it calls this job which operates on a per chunk basis. Since each chunk is organized such that all population entities must be of a single country, the batch.Count means that is the local amount of citizens within that chunk.

However, there may be more chunks containing citizens of the same country being added in parallel in different threads, so Interlocked.Add must be used.

Interlocked.Add() requires a direct pointer to the integer being added to but GetComponentDataFromEntity only returns a temporary copy of Census when read, not a pointer to the actual component.

Thus I needed a pointer Ptr in the component Census that can be read pointing to the actual numerical value storing the census data. Which I've done.

[robust scaffold]

public unsafe struct Census : ISystemStateComponentData, IDisposable
{
    // ReSharper disable once Unity.RedundantHideInInspectorAttribute
    [HideInInspector] public int* Ptr;

    // ReSharper disable once Unity.RedundantAttributeOnTarget
    [SerializeField] public int Value => Ptr[0];

    public Census(bool i)
    {
        Ptr = (int*) UnsafeUtility.Malloc(1, 4, Allocator.Persistent);
        UnsafeUtility.MemClear(Ptr, 1);
    }

    public void Dispose()
    {
        UnsafeUtility.Free(Ptr, Allocator.Persistent);
        Ptr = null;
    }
}```

And it works perfectly fine. Definitely not usable if vectorization is required but there are only about 4000 countries compared to the 1.5M citizens. It's a neat trick to allow for parallelization when conventional methods dont work.

[robust scaffold]

Yea. If your game runs okay, dont try to overhaul it with DOTS.
DOTS should be in mind from the first method call all the way to the end. Trying to patch it into an already existing game is a path that only leads to pain and suffering.
DOTS is also intended for simulation. Think tens of thousands of same prefab GameObjects, maybe with different positions or rotations but otherwise identical. If you use any less than that, stick with conventional GameObjects.

[rugged bridge]

Gotcha, thanks. I do not have a game ready, was trying to learn if I'll need DOTS but seems like I won't, so I'll stick to simpler stuff :p

[robust scaffold]

DOTS is fairly simple, just the construction and coding of it is completely perpendicular to the usual object oriented class and GameObject based coding. I still highky recommend learning DOTS starting up a new project as ECS structure is the way forward if you need any level of performance since it codes "compiler friendly" rather than "human friendly" that Object Oriented is.

I've said it before, DOTS is basically CPU side shader coding. If you pretend that DOTS is a shader that is a lot more flexable with custom defined pixel data, you can go very far with it.

The "entity" is your texture / UV coordinate, the "component" is the pixel data / color, and "system" is the shader.

[amber flicker]

I see. Btw I would definitely expect IJobChunk to be either faster or equivalent depending on batch size. From what I previously read from Joachim, he's of the belief that fundamentally interlocked is never a good idea basically so I'm curious. It's an interesting exercise to think about how to do this in a fast way.

[amber flicker]

Even with 1.5mil entities, I'm still slightly surprised that parallel is much of a win over Run()

If you get a chance, I'd be curious how fast or slow the dumb main threaded solution is. E.g. (psuedocode)

[BurstCompile]
    public struct Test : ISystemBase
    {
        EntityQuery eq;
        public void OnCreate(ref SystemState state) => eq = state.GetEntityQuery(typeof(Something));
        public void OnDestroy(ref SystemState state){}

        [BurstCompile]
        public void OnUpdate(ref SystemState state)
        {
            var censusCDFE = state.GetComponentDataFromEntity<Census>();
            var chunks = eq.CreateArchetypeChunkArray(Allocator.Temp);
            // better to use persistant and resize instead
            var countries = new NativeList<Entity>(chunks.Length, Allocator.Temp);
            var totalsPerCountry = new NativeList<int>(chunks.Length, Allocator.Temp);
            for (int c = 0; c < chunks.Length; ++c)
            {
                var country = chunks[c].GetChunkComponentData(Citzenship).Country;
                var countryIndex = countries.IndexOf(country);
                if (countryIndex >= 0)
                {
                    totalsPerCountry[countryIndex] += chunks[c].Count;
                }
                else
                {
                    countries.Add(country);
                    totalsPerCountry[countries.Length - 1] = chunks[c].Count;
                }
            }
            for (int c = 0; c < countries.Length; ++c) censusCDFE[countries[c]].Total = totalsPerCountry[c];
        }
    }```

[robust scaffold]

Give me a sec to connect all the components together then

[robust scaffold]

You dont need the two native list storage arrays, just directly add the count to the component containing the census using censusCDFE

[amber flicker]

You’re right - sorry - rather hastily written

Was originally thinking of doing that part in a job but then subsequently thought… shrug… if it’s an Isystembase maybe it’s just fast.

[robust scaffold]

Hrm, probably shouldnt use the pointer implementation.

Then again, CDFE does not allow for vectorization anyways

well, lets see if it works

[BurstCompile]
public unsafe void OnUpdate(ref SystemState state)
{
    var censusCDFE = state.GetComponentDataFromEntity<Census>();
    var citizenship = state.GetComponentTypeHandle<CitizenshipChunk>(true);
    var census = state.GetComponentTypeHandle<Census>();

    var countryChunks = _countryQuery.CreateArchetypeChunkArray(Allocator.Temp);
    var popChunks = _popQuery.CreateArchetypeChunkArray(Allocator.Temp);

    foreach (var targetChunk in countryChunks)
    {
        var censusArray = targetChunk.GetNativeArray(census);
        for (var i = 0; i < censusArray.Length; i++)
            censusArray[i].Ptr[0] = 0;
    }

    foreach (var targetChunk in popChunks)
    {
        var countryEntity = targetChunk.GetChunkComponentData(citizenship).Country;
        censusCDFE[countryEntity].Ptr[0] += targetChunk.ChunkEntityCount;
    }
}```

There ya go, forgot the clear section before the summation

[robust scaffold]

Now using the benchmark of about..... 2.1 million pop entities and around.... 600ish chunks. Once this thing loads, I'll give you more exact numbers.

568 countries and 2,338,000 citizens

For a total of 14 country chunks and 2151 citizen chunks

[robust scaffold]

It bounces around quite a bit but it's around 0.5 ms

[amber flicker]

Sorry just sat down to eat - how does that compare?

[robust scaffold]

Compared to the interlocked parallel version displayed there

so about 0.5x longer computational time

and ~10x longer observed time

I can not into math

[robust scaffold]

In terms of strict cycle cost, multithreaded interlocked requires 2x more computational time than mainthread bursted implementation. However, when taken as observed FPS, it's a 90% reduction.

[amber flicker]

Hang on that code above looks quite different to what I expected. Will take a look again in a bit

[robust scaffold]

I'm using my way of directly adding to the CDFE. However, that requires clearing the previous tick's census data so it doesnt keep adding into infinity. So that's what the first ForEach loop is for.

Otherwise, other than the pointer allowing for direct addition rather than a temporary value it's the same as your psuedocode.

If I were to remove the pointer, i could reinterpret the census array of the first ForEach loop to become a int and vectorize set every value to 0. Although I doubt that's the major contributor to the computation time. Vectorizing the first ForEach loop would probably cut down time by at most 0.05 ms and I really dont want to start changing component structure outside this test system

[unkempt abyss]

I've got a memory corruption issue where when I allocate a new NativeList with a temp allocator from inside a job, data inside of an UnsafeList appears to get overwritten by the allocation - only happens when Burst compilation is actually enabled

In both cases, it's a NativeList of UnsafeLists of float3s - so I'm unable to change the UnsafeList to a native list. Basically.... anyone got any bright ideas as to how I'd even begin to debug this?

Code in question:

output:

[robust scaffold]

You can not nest containers in NativeX. If you wish to have a List<List<float3>>, you need Unsafe<Unsafe<float3>>.

Im shocked that the burst compiler didnt complain.

[unkempt abyss]

It's a NativeList of UnsafeLists - are you telling me I need an UnsafeList of UnsafeLists instead? 🤔

[robust scaffold]

Yea. NativeList is a class based implementation of Unsafe with safety checks and other managed things that may mess up the underlying buffer. Also, try to avoid nesting containers. That prevents alignment and vectorization (aka bad performance).

Just change all mentions of NativeList to UnsafeList and see if it works.

[robust scaffold]

Wait, that might not be the problem

Think about what's going on here. NativeList is allocating an array of int* in the temp buffer. This is checked against another NativeList to prevent the pointers from aliasing. However, the pointers within each list are pointing towards another pointer (the second UnsafeList) which is not checked for aliasing