archived-dots

[hollow jolt]

gonna copy this method

do i need to use BurstCompatible for all my methods ?

[frosty siren]

I don't know, and want to figure out too, because I have a lot of utils methods in my project

[hollow jolt]

says that :

Documents and enforces (via generated tests) that the tagged method or property has to stay burst compatible.

seems like its only for docs ...

[hollow jolt]

well i had to make an assembly def with unsafe code to add this method , but it still looks a bit messy in the end :

var data = job.Vertices.AsArray().Reinterpret<Vector3>().ToArray();

NativeList<float3> ==> NativeArray<float3> ==> NativeArray<Vector3> ==> Vector3[]

[hollow jolt]

why JobHandle.ScheduleBatchedJobs( ) negatively impacts performance ?

i want to avoid using Complete( ) and instead listen to jobHandle.isComplete to avoid limiting the executing for a single frame

isn't that what ScheduleBatchedJobs suppose to do ?

or is it the case when calling Job.Schedule( ... ) will auto start it and then i can check for the isComplete value ?

[pulsar jay]

Thats interesting. How did you find this post? There is not even an index on his blog 🤔 The only thing you can find from the home page is the Unite Keynotes.

[haughty rampart]

There's already been some talk about jobs here in this channel.
1- all jobs are limited to run in 1 frame (currently)
2- no job is executed before calling .complete(), only queued up aka scheduled

[haughty rampart]

Well being a student at a private university has its perks I guess. One of my lecturers linked it to a course.

[hollow jolt]

lol was totally not expecting that

kinda defeats of purpose imo

but it does kinda work , just tested it , the frame rate is solid , but the mono behavior that schedules the jobs is slowed down , very odd

[haughty rampart]

Not really. Job system is built for runtime after all. Though lots of people want long running jobs as well. So maybe we'll get them at some point

[hollow jolt]

thinking to set start & end index and schedule the jobs sparsely across multiple frames , it takes some time to build terrain data

is burst compatible with System.Threading ?

[viral sonnet]

no

[hollow jolt]

sad

well time to wack and hack the Jobs across Update calls

[viral sonnet]

your job takes longer than 4 frames? maybe you should operate on Allocator.Persistent data

[hollow jolt]

yea was planning to

[hollow jolt]

^ was rly under the impression that this will not slow down the main thread

[viral sonnet]

schedule will not block the main thread unless you wait for jobhandle.complete

[hollow jolt]

how do i avoid waiting for it ?

that is like my main question , how to use isComplete

[amber flicker]

Not sure where "all jobs are limited to run in 1 frame" comes from. You can just google long running jobs for more info - here's one post from Joachim about it https://forum.unity.com/threads/scheduling-long-running-jobs-from-systems.629347/#post-4216513

[viral sonnet]

you could use a NativeReference<bool> to write your state back. or int and write back the amount already processed and compare that in main thread

[hollow jolt]

For long running jobs i think you must use old school threading. In Unity jobs must start and end in the same thread.

not using Entities

[viral sonnet]

i've not used this ever, maybe the jobhandle is smart enough to give you the isComplete state

the information from GilCat is wrong though

[hollow jolt]

void Update()
{
  if ( ! scheduled )
  {
    scheduled = true;
    
    job = new ... 

    handle = job.Schedule( job.length, default );

    JobHandle.ScheduleBatchedJobs();

    StartCoroutine( AwaitComplete() );
  }
}

System.Collections.IEnumerator AwaitComplete()
{
  while( ! handle.IsCompleted ) yield return null;

  handle.Complete();

  job.Dispose();

  scheduled = false; // *
}

this is what i tried

[hollow jolt]

eh ? so there is a way ?

the code above will slow down the main Update & drop FPS

[haughty rampart]

aaah, please don't mix coroutines and jobs. that hurts just to look at

[hollow jolt]

well how else can i await for it ?

[viral sonnet]

you don't need a coroutine, same could be checked in Update. but that's not the issue here

[hollow jolt]

^

[viral sonnet]

well, you need to find out where it actually blocks your main thread. 2 things you don't need "JobHandle.ScheduleBatchedJobs();" and "handle.Complete();"

[hollow jolt]

i call Complete because :

Tracing data ownership requires dependencies to complete before the control
thread can use them again. It is not enough to check JobHandle.IsCompleted.
You must call the method JobHandle.Complete to regain ownership of the
NativeContainer types to the control thread

[viral sonnet]

let the worker system decide for itself. and the job will be complete at that point so you don't need the complete method

[north bay]

Yep you need to call complete

[viral sonnet]

ok, didn't know that actually

[north bay]

And if I remember correctly ScheduleBatchesJobs is also required to tell the jobsystem to start your job

[hollow jolt]

^

[viral sonnet]

to start it faster, right?

[hollow jolt]

in the docs says that using ScheduleBatchesJobs is actually slower

but i don't want to Complete it on the same frame , aka i don't want to freeze the frame

[north bay]

The job system intentionally delays job execution until you call ScheduleBatchedJobs manually because the cost of actually waking up worker threads can be expensive. Thus a good default is to delay the actual kick until a few jobs have been scheduled. Generally if you are scheduling a bunch of jobs in a loop, wait with kicking the jobs until the end of the loop. If you do significant amounts of work on the main thread between scheduling jobs, then it can make sense to ScheduleBatchedJobs between each job.```

[viral sonnet]

interesting, is this only true for scheduling jobs in monobehaviour? I think I've only ever scheduled jobs in systembase

[devout prairie]

just as i noticed you mentioned terrain - any idea if terrain collision works with physics? i'm using some code workaround someone posted a while back not sure if the implementation in current physics does actually work

[north bay]

SystemBase automatically does this after your OnUpdate

[viral sonnet]

ah makes sense. good to know 🙂

[hollow jolt]

im using Mesh Collider

the "Terrain" is my marching cubes algorithm

[devout prairie]

ah so no terrain object

[hollow jolt]

its not the actual terrain , yes

[devout prairie]

i think this method is quite slow.. i just grabbed it and threw it in quickly as i didn't want to spend much time.. it does work at least

i have noticed i think terrain collider mentioned in the physics namespace but not sure if it actually works

[hollow jolt]

so i commented out JobHandle.ScheduleBatchedJobs(); and looks like u r right , there is no need to call it , Schedule is enough to kick start it later on

[hollow jolt]

which method ?

[devout prairie]

this is interesting and i take it could be used to successfully spawn/instantiate a lot of objects without the lag spike..

[devout prairie]

i'll see if i can dig it up

[devout prairie]

this is the core method, it uses a system to parse the unity terrain into this:

i'll try dig up the original thread

[hollow jolt]

ah ecs-physics , lovely

[devout prairie]

yeah.. are you using havok or something?

[hollow jolt]

wasn't aware the Unity.Physics.TerrainCollider is a thing

[hollow jolt]

no im using the build in PhysX , but i do know a bit about Unity.Physics ( not havok )

[devout prairie]

physx gives you enough performance?

[hollow jolt]

idk yet lol ( didn't compare those two )

its just a terrain , so couldn't be that bad

[devout prairie]

tbh i basically just ran with unity.physics as i was venturing into converting chunks of my game over to dots and was testing out the Rival dots character controller, which uses unity.physics

[hollow jolt]

ah fair enough , in any case last time i checked ecs-physics wasn't actually multi threaded

[devout prairie]

i've not encountered any real problems so far, even managed to get ragdolls working so seems okay

[devout prairie]

yeah i'm not sure i do have raycasts running from parallel jobs but i need to double check at some point if they're actually running in parallel

not sure regards the core physics worldbuilding itself if it's running parallel or not

[hollow jolt]

it just seems to me like its too much hassle to get simple stuff to work with it

also check out this post https://github.com/Dreaming381/Latios-Framework/blob/master/Documentation~/Psyshock Physics/README.md

[devout prairie]

Yeah i have looked at that

it does seem to have some solutions geared towards the networking side of physics, i've not really got to that yet

[devout prairie]

i've not found that tbh

[hollow jolt]

Simulation callbacks are single-threaded - is a bit off putting

physx is gpu accelerated from what i understand

[devout prairie]

yeah i mean it's always possible to switch back with hopefully not too much pain, but so far it's been fine

[hollow jolt]

switch back sound painful

[devout prairie]

Hehe, i do try to write code in a way that makes that possible as much as i can

try not to scatter raycasts around etc, maybe even limit them to one system, things like that

references to colliders in my code are kinda minimal, only really setting collision layers ( filters )

but yeah i think i'd require companion gameobjects to use unity physx right

which i'm not doing at all at this point

[hollow jolt]

any ideas how well burst performs on the Quest ?

[devout prairie]

no idea haven't tried deploying to the quest as yet, i do have the quest 1 & 2 tho!

i think i heard it's almost impossible to get onto the quest store

i guess a lot of devs use sidequest for that

[hollow jolt]

lizard man bad

🤜 🤛

[devout prairie]

🦎 ⚔️

[hollow jolt]

ye that's right , we gonna team up and beat him up

[devout prairie]

androids don't feel pain

[hollow jolt]

i might need to do that as well

[hollow jolt]

I'm getting unsupported write mode error

'UVs' not declared [ReadOnly] in a IJobParallelFor job. The container does not support parallel writing. Please use a more suitable container type.

For the following class :

    [BurstCompile( FloatPrecision.Standard, FloatMode.Fast, CompileSynchronously = true )]
    public struct Polygoniser : IJobFor
    {
        [WriteOnly] public NativeList<float2> UVs;
        [WriteOnly] public NativeList<float3> Vertices;
        [WriteOnly] public NativeList<int> Triangles;

the error popups up when i try to schedule like this :

handle = job_polygon.Schedule( job_polygon.length, noiseHandle );

where noiseHandle is a scheduled IJobParallelFor job - does this mean scheduling JobFor and JobParallelFor is not supported ?

[viral sonnet]

do you have race conditions on UVs? if not, use NativeDisableParallelForRestriction

[hollow jolt]

wdym race condition

[viral sonnet]

do you write to the same index from multiple threads?

[hollow jolt]

yeah .. i hope it won't , all i do is Add item per index ( its a List )

[viral sonnet]

also you don't need WriteOnly for Nativecontainers

if you add to a list use NativeList<T>.ParallelWriter

but keep in mind to pre allocate

[hollow jolt]

so im guessing this is also bad ?

[viral sonnet]

the readonly only works on the NativeArray. for the others it won't do anything

[hollow jolt]

got it

[hollow jolt]

like this ?

[viral sonnet]

no, you use asParallelWriter when you set the parameter in the job struct

not in the job itself

in the job you then have public NativeList<float2>.ParallelWriter UVs;

new jobstruct { UVs = uvs.AsParallelWriter(), ohter params }

[hollow jolt]

does it mean the parallel writer needs no dispose ?

[viral sonnet]

nope

[hollow jolt]

hmm ok

[viral sonnet]

outside of the job you'd use the normal NativeList

the code you posted is main thread code, right?

[hollow jolt]

ah i understand now

as parallel writer is the reference which is executed inside the job and we can keep the list outside for access , right ?

[viral sonnet]

yes

it's like EntityCommandBuffer.Concurrent if you've ever used that

just a wrapper for parallel jobs

[hollow jolt]

didn't use that , but i see what u mean

[digital panther]

Big big big

[hollow jolt]

there is no info in the docs about how to get ParallelWriter length

[viral sonnet]

you have to set the capacity of the NativeList in the main thread

[hollow jolt]

idk what its capacity in advance , and i need to know the current size per index

ill try using NativeDisableParallelForRestriction

[viral sonnet]

if you exceed the list length it will throw out of capacity

can you find out the capacity you need beforehand? otherwise this job is not fit for parallel processing

[hollow jolt]

when i declare the list i don't set its length

its not a parallel job - IJobFor

[viral sonnet]

do you schedule or scheduleparallel?

if you schedule, then I'm sorry, you don't need a parallelwriter then

[hollow jolt]

Schedule

[hollow jolt]

^

was surprised the error showed IJobParallelFor in the message

[viral sonnet]

yeah, that's why I thought you schedule parallel in the first place 😄

[hollow jolt]

the handle before that is a parallel one

maybe the error message is broken ?

[viral sonnet]

don't think so

can you post full code? otherwise it's confusing

[hollow jolt]

yep 1 min

[viral sonnet]

also possible, the writeOnly tag throws it off. did you try without?

[hollow jolt]

InvalidOperationException: Polygoniser.UVs is not declared [ReadOnly] in a IJobParallelFor job. The container does not support parallel writing. Please use a more suitable container type.

[hollow jolt]

yep its gone , same error

[viral sonnet]

i don't understand the job_polygon.length variable

seems like you don't set it and it's 0?

[hollow jolt]

        public int3 gridSize;
        public float isolevel;

        public int length => gridSize.x * gridSize.y * gridSize.z;

[viral sonnet]

i see

both jobs are IJobFor?

[hollow jolt]

nope the first one is parallel

^

where noiseHandle is a scheduled  IJobParallelFor  job - does this mean scheduling JobFor and JobParallelFor is not supported ?

[viral sonnet]

job_noise? can you try completing it before polygoniser?

[hollow jolt]

ye i already tested it and it works fine on its own

oh u mean force execute it

was kinda hoping to schedule those one after another and check for the handle isComplete

[viral sonnet]

yeah, I don't see anything wrong with the code. something throws it off. scheduling parallel and single should not be a problem

[hollow jolt]

🤔

[viral sonnet]

even the example uses a single then a parallel schedule https://docs.unity3d.com/ScriptReference/Unity.Jobs.IJobFor.html

[hollow jolt]

well i just tried noiseHandle.Complete(); in the middle of the second screen shot

but still the same error

InvalidOperationException: Polygoniser.UVs is not declared [ReadOnly] in a IJobParallelFor job. The container does not support parallel writing. Please use a more suitable container type.

but its not IJobParallelFor ...

[viral sonnet]

what does the profiler timeline show you?

[hollow jolt]

i can't run it

this is burst error

[viral sonnet]

is the polygoniser job single or in multiple worker threads

huh? this is not a runtime error but a compiler error?

[hollow jolt]

ye

InvalidOperationException: Polygoniser.UVs is not declared [ReadOnly] in a IJobParallelFor job. The container does not support parallel writing. Please use a more suitable container type.
Unity.Jobs.LowLevel.Unsafe.JobsUtility.CreateJobReflectionData (System.Type type, Unity.Jobs.LowLevel.Unsafe.JobType jobType, System.Object managedJobFunction0, System.Object managedJobFunction1, System.Object managedJobFunction2) (at <0e7c1c9367c544fa83b5270f5a69cf11>:0)
Unity.Jobs.IJobForExtensions+ForJobStruct`1[T].Initialize (System.Boolean asParallel) (at <0e7c1c9367c544fa83b5270f5a69cf11>:0)
Unity.Jobs.IJobForExtensions.Schedule[T] (T jobData, System.Int32 arrayLength, Unity.Jobs.JobHandle dependency) (at <0e7c1c9367c544fa83b5270f5a69cf11>:0)
Mesh05.LateUpdate () (at Assets/Mesh05.cs:61)

[viral sonnet]

IJob is the same as IJobFor. maybe try that

IJobFor just supports schedule and scheduleparallel

[hollow jolt]

ah ok sounds like that might be it

oh no that would be too slow lol

i need the Execute ( index )

[viral sonnet]

just use a for loop in the execute

[hollow jolt]

then it would be using only a single thread , no ?

[viral sonnet]

ijobfor and schedule is single threaded

i.e. single worker thread

[hollow jolt]

oh

so i tried NativeDisableParallelForRestriction and instead of IJob used IJobParallelFor

no errors ( gonna try run it and see the editor crash 👀 )

[viral sonnet]

with parallel you then have to use the NativeList<T>.ParallelWriter

[hollow jolt]

understood

[frosty siren]

i see that NativeList<T>.PrallelWriter only has AddNoResize(), which means that i need to allocate list with enough capacity? If i has a big source nativearray and i want to filter it, should i use parallel nativelist with capacity equals to nativearray whole length? or maybe use parallel nativequeue and then use ToNativeArray() ?

[viral sonnet]

are you going for max performance? because both are not that great

nativelist.parllelwriter has an atomic operation on the length when adding. nativequeue, I don't know the details, but was also slow

a good alternative for parallel writing is NativeStream

that apart, yes, set the list capacity to the native array length

[frosty siren]

yep, for my particular case i want max performance. Does NativeStream far advanced?

[viral sonnet]

nativestream has the best performance on parallel writing. honestly, parallel NativeContainers are pretty weak right now. The next best thing is: https://github.com/Dreaming381/Latios-Framework/blob/v0.4.2/Core/Core/Internal/UnsafeParallelBlockList.cs

[frosty siren]

hope all this performance treasures will be well documented 🙂

[viral sonnet]

his blocklist has problems with keeping thread activity balanced when reading though. it all has pros/cons

[frosty siren]

what is foreachCount and why it has cunstruction job? To be able to chain it's construction depending on results of previous jobs?

[viral sonnet]

that would be the amount of elements you'll write

[frosty siren]

is that simply Length?

[viral sonnet]

yes

[frosty siren]

mmm, why it has foreach prefix then? 🙂

[viral sonnet]

and you can't write to the same index with NativeStream

i dunno, it's a weird name, I agree

guess because it was designed to be used in IJobChunk

[frosty siren]

NativeStream.Allocate is how it can be resized but only on main thread?

[viral sonnet]

you don't need resizing, it's based on block allocation

allocate is only needed if you want to get a ref element

[frosty siren]

i don't familiar with term "block allocation". Is it like in simple all arrays/lists, when we have shifted elements in memory, or it something different?

[viral sonnet]

here's the real allocate

every thread gets a block of memory to write to threads don't block them self from writing

this block can fit a number of elements

so not every write is an allocate

[frosty siren]

oh, ok, that is clear. Should i define blocks by myself, or it goes automagically?

[viral sonnet]

I think lots of NativeContainers were hastily written. NativeStream uses hardcoded 4k blocks

can't be changed

even uses a const for size that isn't used

[frosty siren]

well, if i'll allocate NativeStream with 0 foreachCount and will use Writer.Write(), will the block be resized?

[viral sonnet]

resized is the wrong word. a block of 4k will be allocated which then can be filled

[frosty siren]

yes, i mean that 🙂

so with NativeStream i have no limit like with NativeList, where i need to know capacity before write.
is it how NativeQueue works? Like each thread uses own block and allocate block on need?

[viral sonnet]

when you use NativeStream as intended with the foreachcount inside a job that has the same elements all is well but if you use another range it'll throw an error that the bounds are not correct. that can be disabled with NativeDisableContainerSafetyRestriction

sadly NativeQueue uses a bunch of atomic operations so threads will block each other when enqueuing and allocating blocks

it's quite slow and sadly useless in parallel writes

not sure what they were thinking

I had quite the specific case to write 250k elements in a frame and the UnsafeParallelBlockList and NativeStream were the clear winners. NativeList and Queue were all quite bad

list was worst with an atomic operation on every length add

they are so far away from "performance by default" that I don't even understand why they bothered adding a ParallelWriter. It just confuses the dev, expecting that this has been thought through

a single threaded write will be faster than a parallel one. performance tanks at around 5x

[frosty siren]

have anyone asked them about improving of this things in 0.5 - 1.0 updates?

at least we can expect supporting last version of collections in 1.0 update, and you've mentioned that there is UnsafeParallelBlockList in future versions.

[viral sonnet]

the UnsafeParallelBlockList is just something that DreamingImLatios wrote. It'll probably not be added to the collections

And yeah, I wanted to ask them if there are will be any improvements to the native collections. But tbh, I feel like I'm in a position that I just trash everything they are doing. lol

maybe someone else wants to step up 😄

[frosty siren]

lol, i was sure you was refering unity's repo

[frosty siren]

i think, if adult devs can filter theirs emotions then criticism helps form sober view

again there was always much love and trust to unity and there is already cult of dots, so today a hard word, i think, is not excess

[viral sonnet]

probably true, yeah, I should write a thread about it. I think all NativeContainers with Concurrent or ParallelWriters should be non-blocking

When it's impossible, make the dev aware. Like NativeList.ParallelSlowWriter 🙂

do you know the NativeCounter example?

[frosty siren]

no, but have seen mentions a couple of times

[viral sonnet]

interesting read: https://docs.unity3d.com/Packages/com.unity.jobs@0.1/manual/custom_job_types.html

single threaded, 1 int. multi threaded, job count amount of ints. that's basically how every non-blocking nativecontainer would be designed

[robust scaffold]

The forums are currently on fire, absolutely hilarious.

[viral sonnet]

how so?

[frosty siren]

do you mean activity of asking questions or brand new wave of argues about dead/alive status?

[safe lintel]

probably referring to how as you observed, the inability for devs to filter their emotions from their questioning

[frosty siren]

please notice that i mean not all in forum 🙂

[safe lintel]

i know, a minority, though they tend to make the most noise 😅

[robust scaffold]

Yea. The latter. People saying it's not dead, dead, should be dead, undead.

[frosty siren]

what am i doing wrong with reading of NativeStream?

var count = renderArchetypeTransitions.Count();
var transitionsReader = renderArchetypeTransitions.AsReader();
str += $"\ntransitions ({count})\n";
for(int i = 0; i < count; i++)
{
  transitionsReader.BeginForEachIndex(i);
  str += $"{transitionsReader.Read<int>()}\n";            
  transitionsReader.EndForEachIndex();
}

Debug.Log(str);

What i get is below, and after element with '6' value i get error: There are no more items left to be read.

//transitions (7)
//1
//4
//6

Oh, ok, i see now, that NativeStream should be read per-buffer. Reader.ForEachCount is buffer count and RemainingItemCount is how much to read in current buffer

[devout prairie]

any idea if it's possible to move/rotate an existing collider in unity.physics?

[viral sonnet]

yes, transitionsReader.BeginForEachIndex(i); returns the count to read

[frosty siren]

seems its not

sorry, it was writer 🙂

[broken cradle]

is there a way to reallocate a persistent native array ?

[frosty siren]

dispose and then allocate?

[broken cradle]

i want to keep the values that are already in there

so no

i could make a second one, copy with a for loop

but that seems slow i would rather have a reallocation

[frosty siren]

allocate new one -> use NativeArray.Copy -> Dispose old one

[broken cradle]

ok so no reallocation i suppose

i'll go for the copy then

[frosty siren]

how should i figure out what index to use when execute parallel job func with NativeStreamWriter? Also i see, that foreachCount in NativeStream constructor seems to be buffer count rather than element count

[viral sonnet]

it's a form of index that prevents that 2 threads write to the same index

that's pretty much all there is to it

you could use [NativeSetThreadIndex] as BeginForEachIndex value

[frosty siren]

so i need to get thread index

[broken cradle]

think of it as an array index, the parallel for execute your code for each element at index i

[viral sonnet]

or you use the entity index

what's great about it, is that you can write more than 1 element to 1 index from the same thread. as long as they don't block each other it's okay. otherwise you will get an error msg

[frosty siren]

this part i understand

[broken cradle]

i donj't really get how you can have a filestream with a parallel for, isn't there some documentation about this ? Does it work per line ?

[frosty siren]

i think i can't figure out something very simple 😬
[NativeSetThreadIndex] injects thread index. If i use it in BeginForEachIndex in Execute func in IJobParallelFor then in the same thread i will call this func N times, which causes an error BeginForEachIndex can only be called once for the same index (N)

Is BeginForEachIndex() starts writing process to block or to element? Sorry for this noob questions attacking

[cerulean pulsar]

Hi, I'm trying to cast one collider against another in Unity.Physics, but using colliderA.CastCollider(ColliderCastInput input) requires the translation of colliderA to be baked into the collider data of colliderA. Is there some alternative where I can provide a translation of both colliders, like I'm able to provide a translation for colliderB via ColliderCastInput?

[viral sonnet]

you only need BeginForEachIndex() once per index

or per Execute

if you use threadINdex

[frosty siren]

i was going to ask: is it absolutely ok? I mean, you've mentioned that one block is 4k, and having N * 4k where n is num of entities is bad? or this BeginForEachIndex is not about blocks?

[viral sonnet]

it's not about blocks, that's under the hood

don't worry about the blocks or its size

just write what you want to a given index

[frosty siren]

i know i can use IJobParallelForWithBatch which has extra method which called per batch but only before executes, so how then call EndForEach ?

[viral sonnet]

can you show me your for loop?

[frosty siren]

sure

private struct FindRenderBatchesTransitions : IJobParallelFor
{
  [ReadOnly]
  public NativeArray<SortingData> sortingDataArray;
  public NativeStream.Writer transitions;
  //[NativeSetThreadIndex]
  //private int _threadIndex;

  public void Execute(int index)
  {
    transitions.BeginForEachIndex(index);
    if(sortingDataArray[index].archetypeIndex != sortingDataArray[index + 1].archetypeIndex)
      transitions.Write(index);
    transitions.EndForEachIndex();
  }
}

[viral sonnet]

looks good, where's the issue?

[frosty siren]

no no, no issues, i just looking for possible performance problems or possibilities to improve, because i'm not familiar with all this native stuff

[viral sonnet]

ah ok, yeah that's how it's used.

all good! 🙂

you can measure performance and/or could compare wit nativequeue, etc...

[frosty siren]

for now i'm trying to build something that can at least work without errors 😅

[viral sonnet]

it's pretty much all there is to nativestream. if there are any advanced usages, i don't know them either 😄

alright, I think that's the fastest you can get anyway 🙂

well apart from having totally perfect pre-allocated memory

but that's sometimes not really possible

[frosty siren]

var renderArchetypeTransitions = new NativeStream(sortingDataArray.Length, Allocator.TempJob); here i allocate NativeStream with length of whole not filtered array. Is it ok, or should i remain length be 0 ? Asking because i've tried with disabling safety checks with attr and errors was still throwing

[viral sonnet]

from your use case though, I think a nativearray would also be alright. the overhead of setting it to something like -1 or memclear could cost a lot though so that would need testing

init the nativestream with the same length as the FindRenderBatchesTransitions job

seems like sortingDataArray.Length to me

ah, yes, what you have posted lol

[frosty siren]

can make things more complex and use IJobParallelForBatch which allow me have less foreachCount equals to thread count

[viral sonnet]

the fewest counts would be with using the threadIndex

give it a try wit batching first

[visual tundra]

This is anything to do with them being 4 years into DOTS whilst seemingly 4 years from finishing and everything else having been neglected during that time?

[unborn totem]

Hello folks. Is there a way to copy all component data from one entity to another already existing entity that works inside of Burst?

So far I have only seen a way to copy a specifically known component's data.

[hollow jolt]

is it safe to use int in IJobParallelFor ? something like this :

struct A : IJobParallelFor
{
  public int count ;
  public NativeStream.Writer output;
  
  public void Execute( int i )
  {
    // do something with output .. 

    if( ... ) count ++ ;
  }
}
...
void main()
{
  new A() { ... , count = 0 }.Schedule( length, 8, default );
}

[hollow jolt]

is there a way to subscribe to a on complete event from the first Job ? The way i have it scheduled right now is wrong so that's an example of what i want to archive without calling handle . complete () ( since the first job will take several frames to complete )

[pulsar jay]

You need to add a depandency on the first job to the second job. Dont remember what the method was though. Maybe just AddDependency 🤔

[hollow jolt]

no but i need to have outputCount as a result of Job1 execution to know how much memory to allocate for Job2 ( array size )

Job2 . schedule ( handle1 ) - already combines dependencies

[pulsar jay]

yes you are right. I overlooked it. Afaik you need some kind of native container to share data between jobs. even if its just a NativeArray with size 1. I dont have much experience with using job dependencies though...

[hollow jolt]

the steam is a native container which is shared between the two jobs

i need something like callback event to listen to when job1 is complete

so i know how much memory to allocate for job2

( look at the arrow above , job1 will define the value of outputCount , job2 will use that value to allocated array with that length )

[amber flicker]

You may be looking for AsDeferredJobArray() - I've not used it myself and there aren't many examples but the docs have one: https://docs.unity3d.com/Packages/com.unity.collections@1.1/api/Unity.Collections.NativeList-1.html?q=nativelist

[hollow jolt]

in my case outputCount has no correlation with the stream length so even if i use native array - that won't help making it deferred

[amber flicker]

ah sorry, I just took a better look at your code. For that case I would just use a persistent list... though I'm not sure that totally solves it for you

[hollow jolt]

what's a persistent list ?

[amber flicker]

just use NativeList<...> = new .... Allocator.Persistent

and keep it in the system instead of the job

[hollow jolt]

yeah its one way to solve it i guess removing fixed arrays ...

[amber flicker]

nativelist can be iterated like arrays (you can use .AsArray()) and it'll be much faster to have one list that resizes than creating a bunch of Temp arrays

[hollow jolt]

really ?

[amber flicker]

what's surprising about that?

[hollow jolt]

idk seems like list changes size , where an array is fixed that should be faster , no ?

[amber flicker]

if the job only ran once then maybe there'd be a difference but as soon as you run it more than once, instead of allocating new memory, you're either reusing the memory you've allocated or increasing the allocation by some amount. That allocation (especially TempJob) is quite expensive.

ideally if you allocate the list with your most likely maximum capacity, you may never need to allocate for that container again

[hollow jolt]

hmm true

target.Clear();       /// Mesh 
target.vertices =     J2.out_ver.Reinterpret<Vector3>().ToArray();
target.uv =           J2.out_uvs.Reinterpret<Vector2>().ToArray();
target.triangles =    J2.out_tri.ToArray();

i think 3 arrays would be faster in this case

[amber flicker]

hmm I wonder whether I'm missing the forest for the trees here - are you generating meshes? Are you aware of the more recent api's? Not sure you need to be using Vector3s etc

[hollow jolt]

which api ?

and yes im generating meshes

[amber flicker]

first google result related to what I'm talking about (not necessarily the best reference) - https://forum.unity.com/threads/parallel-procedural-mesh-generation-with-job-system.959385/#post-6379971

[hollow jolt]

im not using ECS

[amber flicker]

I just meant e.g. the var dataArray = Mesh.AllocateWritableMeshData(1); syntax

[hollow jolt]

is this in 2020 ? ( my current project is 2019 )

[amber flicker]

oh 🙂

2020 only I think yup

[hollow jolt]

interesting ill try that

[amber flicker]

Fyi these allocations (ToArray) may then become a bit of a bottleneck. I don't know your use case but say you were generating 100 meshes with similar sizes, again allocating once then resizing or using a subarray can be quicker. So some Vector3[] verts; then iterating J2.out_ver and setting each element in verts can actually be quicker.

[hollow jolt]

interesting , so its a trade of memory for computation power

looks like i would need AllocateWritableMeshData each frame regardless since it needs to call ApplyAndDisposeWritableMeshData - any idea if it can be made persistent somehow ?

[amber flicker]

if you're making new meshes, yes you'll need to allocate memory for each new mesh

[hollow jolt]

what if im not making new meshes ?

[amber flicker]

well... that would probably be a bit unusual I think as I guess it would mean you were making multiple meshes but only ever uploading one to the gpu for rendering - is that what you're doing?

[hollow jolt]

no im making one mesh per job

and then i want to edit it and update the vertices during gameplay

so only 1 mesh would be updated in realtime , the rest would be out of reach for the player

its a marching cubes terrain algorithm

[amber flicker]

I don't know the most efficient way to approach that but it sounds like something that would have been done quite a few times before

[zenith wyvern]

You don't need the ApplyAndDispose api for what you're doing. You're better off using persistent nativearrays. You keep all your necessary mesh data in nativearrays/nativelists (float2 uvs, float3 vertices, etc) then push them to the mesh on the main thread using the mesh.SetVertexBuffer api and using list.AsNativeArray() at the copy site. This copies your native array data to the gpu without causing any managed allocations

https://github.com/sarkahn/dots-roguelike/blob/master/Assets/Lib/Terminals/Runtime/Rendering/TerminalRenderMeshSystems.cs for an example, the api is (somehow) still not well documented

[hollow jolt]

can we use SetVertexBufferData inside a burst job ?

[zenith wyvern]

Nope. Regardless of what api you're using the moment when you copy your data to the gpu (ApplyAndDispose or SetVertexBufferData) must happen on the main thread

From my testing ApplyAndDispose is only good if you have a huge amount of data that isn't going to be changed a lot. Otherwise it's not worth it to have to dispose the data every time you copy.

[hollow jolt]

does this means AllocateWritableMeshData is better for new meshes and SetVertexBufferData is preferred for dynamic meshes

[zenith wyvern]

Exactly

[hollow jolt]

oh i see , thanks

[zenith wyvern]

At least from my experience that's what I found

[hollow jolt]

ill try use both ( first allocate when chunk is created and then set buffer on change )

.

which container is better to share across two jobs that run one after another ( scheduled , where job1 writes and job2 reads ) NativeList.AsParallelWriter / Reader or NativeStream<T>.AsWriter / Reader ?

something tells me native stream isn't as efficient as it can read / write Any* data type and the memory steps are not fixed

[frosty siren]

hi, there is plenty of threads on forum about this problem, but long story short: we have no built-in tool for that (i sure the archemages of dots can advice some hucky unsafe solution).

[gusty comet]

What's left in DOTS to be complete and be released? Also what is complete?

[frosty siren]

you can read questions and figure out what is not complete 🥲 https://forum.unity.com/threads/dots-development-status-and-next-milestones-december-2021.1209727/

[hollow jolt]

can't seem to find how to write triangle ( int ) indexes , does it mean instead of building my triangle array i need to sort the vertices ?

[zenith wyvern]

mesh.SetIndexBufferData(indices, 0, 0, indices.Length, Flags);

Also note the mesh.SetIndexBufferParams(6 * tileCount, IndexFormat.UInt16);. U16 means ushort.

[hollow jolt]

ah so this is mesh.triangles

[hollow jolt]

why is your SetVertexBufferParams & SetIndexBufferParams have different lengths ?

[zenith wyvern]

Because a quad is described by 4 vertices and 6 indices

[hollow jolt]

true

[safe lintel]

@gusty comet so much yet to be completed

[gusty comet]

how can i rewrite my code in dots

im trying to grasp the whole idea

[coarse turtle]

what are you trying to convert to a more data oriented way?

[gusty comet]

i want to increase the performance

[coarse turtle]

okay have you profiled your game and looked at what is pretty slow at the moment?

If you're new to DOTS and need a few examples on how it's written you can take a look at the EntityComponentSystemSamples: https://github.com/Unity-Technologies/EntityComponentSystemSamples
(They're decent enough to get started with before you start doing a deeper dive into DOTS).

There are a few resources pinned in this channel too

[frosty siren]

anybody know what with NativeStream.ToNativeArray performance? Is it ok to use with large set of elements?

[viral sonnet]

well it's single threaded and a struct memcpy

so for a large amount of elements it's not great

it's best to read it in parallel with a ref var on the Read<T>

[frosty siren]

i'm trying to implement algorithm which will find breaks of equals numbers in a sequence, and maybe i just can't find suit approach, but 1st i look those breaks in a way like for each individual index from 0 to N - 1 look to right neighbor and check if value is not equal -> if yes then it is break and write index to stream.
Then for each detected break i want to find previous break to define index range like "there is value X from N to M index in a sequence".
And also i need to write 1 extra break index in main thread, because groups is always breaksCount + 1 length.

[gusty comet]

"breaks of equals numbers in a sequence"?

[unborn totem]

thanks Tony

[frosty siren]

maybe it is my bad eng. I mean when you have sequence of 111112222, then i want to detect break on 4 index and then get 2 groups (0, 4) and (5, 8)

[gusty comet]

"Then for each detected break i want to find previous break to define index range like "there is value X from N to M index in a sequence". => Easy because you already have all "break" start values. So you know the start and end of each group (end is start of prev -1). And just sample first in the "group".

For performance save the break start indexes in an array. And depending on when and how you need to values then save the sampled values in an array also.

Pass the arrays to a readonly variable available in the main threads after each bach (or whenever you need).

[frosty siren]

that is how i'm writing it now

[gusty comet]

Roger. So you are asking or a better algo? Sounds fine to me.

*for

[robust scaffold]

From a recent communication from Unity DOTS lead dev:

On the second question: Clearly ISystem implementations cannot easily access the managed part of the world, but most of the guts live in an unmanaged struct, and you can query that unmanaged part of the world for other systems from the OnUpdate-callback in your ISystem. Essentially, the managed part of the world is just forwarding to the unmanaged part in most cases, so almost all operations are available. In general, we'll do our best to make ISystem not just on-par with SystemBase but hopefully even better (like, you know, burst compile the thing ;)).

Interesting. Clearly some work has been done with ISystemBase. Hopefully it can support all job types. Then I'll just add an "I" to all my systems and then call it a day (along with renaming the method headers). But good news all around. Release date (my guess) March 2022.

[visual tundra]

A mission decision is missing. Everything else that followed is a direct result of that absence.

[robust scaffold]

The decision makers came directly from the C Suite, the co-founder of Unity himself Joachim Ante (CTO) is leading this drive. The radio silence from Unity during the past year was definitely an executive decision by him.

[visual tundra]

Cast your mind back to when the DOTS name was created. Now cast back further, and remember the pitches before then. Changing directions this radically, thrice, means there's been (and remains) no mission decision.

[robust scaffold]

Oh yea. Hrm. I jumped on the bandwagon back when everything was still called Unity ECS and there were a lot of grumbling about the recent transition from [Inject]. The tutorials all used [Inject] so getting started was an absolute pain. Now even before that, my personal history is a lot more blurry.

[robust scaffold]

And no concrete mission means development hell. Which is what is going on with DOTS. Fun.

@viral sonnet

@Kmsxkuse I've just had a quick look and I think that 1.7.0-pre.2 (due sometime in January'22) should have a fix for that inspector issue.
From the Burst team. They fixed the inspector issue but the fix will land sometime next month. Feels amazing to have an actual ETA for a fix *cough* DOTS.

[visual tundra]

Similar experiences. I remember being enthused by them hiring Mike Acton. Then, within less than 2 years, seeing him give a speech on why we should all blame ourselves. Democracy in Act(i)on.

[viral sonnet]

Imagine having a team of developers that can give ETAs. The only good thing in DOTS right now is Burst. The technology and the team is amazing. Everything else ... pretty meh

[robust scaffold]

My post is liked by a burst dev. My life is complete.

[viral sonnet]

just look at him, he knows he's great

now if some of this talent could go over to the other dots projects ...

[whole inlet]

Is there a way to find and serialize all components on an entity? Or some way to essentially save the game state to a file and restore?

[robust scaffold]

Not... really. Nothing built in. I've been using NewtonsoftJson to read and write to .json files for every individual entity for logging and saving/restoring purposes. Of course, this cant be done in parallel nor can it be done with Burst but in a single thread IJob, sure.

[zenith wyvern]

There's https://docs.unity3d.com/Packages/com.unity.entities@0.17/api/Unity.Entities.Serialization.SerializeUtility.html

[robust scaffold]

The files it produces also number in the thousands so I also zip the resulting folder then stream unzip when I reverse it.

[zenith wyvern]

Managed components require manual fiddling, but if everything is purely unmanaged it kinda "just works"

Assuming you're not worried about versioning

[robust scaffold]

Serialization does not work with Chunk components. It's unreliable for shared component data. Shared component data with an entity in it will not properly remap to an entity.

[zenith wyvern]

Yes, you have to use the remap utility for things like that

[robust scaffold]

Chunk components as a whole are not supported by entity remap.

[viral sonnet]

chances are you don't need eveyr single data in the world and entities saved. just write the relevant IComps/buffers to a struct and json serialize it. easiest solution I think

[whole inlet]

Yeah i'm currently using newtonsoftJson to serialize but i'm finding it hard to deserialize back to the proper system types

[robust scaffold]

This is what I do to assign shared component data containing entity fields.

As for chunk component data, I have a system that mirrors shared onto chunks using a IJobEntityBatch. Main thread and not burst compiled of course.

[viral sonnet]

why? create the entity archetype and set component from the save game.

[robust scaffold]

private struct MirrorSharedToChunk<TShared, TDestination> : IJobEntityBatch
    where TShared : struct, ISharedComponentData where TDestination : struct, IComponentData
{
    public EntityManager EntityManager;

    public SharedComponentTypeHandle<TShared> SharedSource;
    public ComponentTypeHandle<TDestination> ComponentDestination;

    public void Execute(ArchetypeChunk batchInChunk, int batchIndex)
    {
        var source = batchInChunk.GetSharedComponentData(SharedSource, EntityManager);
        batchInChunk.SetChunkComponentData(ComponentDestination,
            UnsafeUtility.As<TShared, TDestination>(ref source));
    }
}```

[whole inlet]

can you give me an example of the "set component from the save game" piece? As to "why" i chose this method, it just happened to be the only way I thought of at the time, happy to change to best practice or a better way

[robust scaffold]

This is what I do for initialization from Json. Not the fastest, about 6-8 seconds on my computer. But it works without locking the graphics / main thread.

Create a new world, pull the exclusive entity transaction from it, run a few jobs off it, then merge the completed entity jobs back into the main world and destroy the loading one.

Now I'm sure it can be much better optimized, especially if I get off my ass and convert the json files to contain pure blittable values (thus burst and parallelizable), but this works

[viral sonnet]

in case json is slow to deserialize, messagepack is pretty cool

[robust scaffold]

Reading the files dont take that long surprisingly. Only 2-3 seconds.

[whole inlet]

thanks everyone for your help, i'm taking a closer look at your recommendations and examples and will implement accordingly!

[hollow jolt]

yeah i don't use ecs for the same reason , looks like Burst + Graphics.Draw / ComputeBuffers is faster then ecs anyway

[robust scaffold]

There are some issues with compute buffers. I dont have experience with it but it's described here: https://forum.unity.com/threads/dots-things-no-one-talked-about-in-2021.1210731/
Primarily:

First, allocating a ComputeBuffer always allocates GC. This means that if we ever what to defragment GPU memory, we have to allocate GC memory on the CPU. That’s really bad, and currently working with ComputeBuffers is a bit of a pain with regards to memory management.

[hollow jolt]

well fingers crossed for 2022

[visual tundra]

What's this Graphics.Draw thing all about? Is it good/fun?

[hollow jolt]

iirc this is whatthe hybrid pipeline is using under the hood

https://docs.unity3d.com/ScriptReference/Graphics.DrawMeshInstanced.html

pretty amazing actually , u can draw millions of meshes with that thing

i did try it last time like 2 years ago https://cdn.discordapp.com/attachments/605501877131542556/755530786488320080/eAdYTWTChR.mp4

[zenith wyvern]

v1 used instancing, afaik v2 doesn't use it, just batching

Instancing doesn't play well when you want to start introducing per instance data on a large scale

[hollow jolt]

how does it render stuff ?

[zenith wyvern]

With the "BatchRenderGroup" api, however that works under the hood

[hollow jolt]

from what i see, use unity as normal , and when in need of a 10k objects , use Graphics + Burst

[hollow jolt]

that's a new one

any example how to use that ?

[zenith wyvern]

Nope. I never tried and it's not well documented from what I've seen

[viral sonnet]

I wonder why Unity still bothers with per frame render data. With compute shaders this could be a thing of the past.

[molten flame]

Anyone using a toon shader that supports skinning here? I tried the latest unitytoon/universaltoon shader here https://github.com/unity3d-jp/UnityChanToonShaderVer2_Project/tree/release/urp%2F2.3.0 that supports SRP batching but it doesn't seem to support skinning..

[visual tundra]

Do you know if this is applicable to Mobile, or only DirectX etc?

[hollow jolt]

InvalidOperationException: The previously scheduled job PolygoniserParallel writes to the Unity.Collections.NativeList`1[MCTerrain.Triangle] PolygoniserParallel.output. You must call JobHandle.Complete() on the job PolygoniserParallel, before you can read from the Unity.Collections.NativeList`1[MCTerrain.Triangle] safely.

Any ideas why this needs to be completed ? i don't understand - though that schedule will already make sure that the first job is complete before the second one start ... ?

public NativeList<Triangle> triangles;

...

var J1 = new PolygoniserParallel
{
    output = triangles.AsParallelWriter(),
};

var H1 = J1.Schedule(parameters.length, 16, dependency);

var J2 = new PolygoniserArray
{
    input = triangles.AsParallelReader(),
};

var H2 = J2.Schedule(H1);

[hollow jolt]

i think that should be fine with vulkan on android ( or maybe this is specific to compute shaders ... )

[visual tundra]

this is EXTREMELY interesting. Thank you VERY much!!!

[hollow jolt]

yea np , some platforms (webGL) might not allow graphics - direct instanced , but u still can use the other methods

[zenith wyvern]

What is MCTerrain.Triangle

Oh just the index

[hollow jolt]

^ for that first line NativeList< struct >

[hollow jolt]

https://docs.unity3d.com/ScriptReference/Graphics.html

[zenith wyvern]

It may be complaining about you re-using the native list on the next frame - you might have to keep the job handle from the previous frame and manually complete it before you pass the list back in to "J1"

[robust scaffold]

.AsParallelWriter() procs the job safety check. Instead use .AsParallelWriter() before J1 to store the parallel writer version in a local variable and then pass that local variable to input.

[hollow jolt]

👀 doesn't make sense , but thank you , ill try that

[zenith wyvern]

Also you don't need a bunch of separate job handles like that, it just makes it confusing. You can do

var dep = job1.schedule(whatever, dep);
dep = job2.schedule(whatever, dep);

And so on

[hollow jolt]

true

[zenith wyvern]

What doesn't make sense?

[robust scaffold]

Wait, reader. There's no such thing as .AsParallelReader()... I think. You can read from a NativeList in parallel using [ReadOnly]...

Either way, store the ParallelWriter and ParallelReader in a local variable before creation and scheduling of the jobs then pass those into the input

[hollow jolt]

good to know

[robust scaffold]

NativeArray<T>.ReadOnly

Yea, it just returns a native array that enforces a readonly check. If you're using struct jobs, use the [ReadOnly] attribute. I believe .AsParallelReader is only used in the Entities.ForEach Lambda gens.

[hollow jolt]

this is my input field : public NativeArray<Triangle>.ReadOnly input;

so i change to " 1[ReadOnly] public NativeArray<Triangle> input and i should just pass the NativeList as a reference ?

[robust scaffold]

You can use [ReadOnly] public NativeList<Triangle> input and then yea, pass in just triangles.

And with that, you dont need to store .AsParallelWriter() in a local variable.

[robust scaffold]

Oh yea, make sure it's NativeList, not NativeArray

[zenith wyvern]

Would any of that affect dependency? It's all pointing to the same list

[hollow jolt]

nope still getting the same safety error

[robust scaffold]

Hrm, I think NativeList may have some issues with vectorization since it's a pointer wrapper around NativeArray. Make sure in your IJob struct to obtain the .AsArray() and then use that native array.

[robust scaffold]

Hrm. I do this all the time though...

[robust scaffold]

What's the new job chain look like?

[zenith wyvern]

Are you assigning your job handle back to the system dependency before OnUpdate ends?

[hollow jolt]

collections package v.14

[hollow jolt]

LateUpdate()

[zenith wyvern]

Oh...then yeah you definitely need to be storing the job handle.

And calling complete manually.

Like I said the first time.

Because you're re-using the same native container on the main thread you need to call complete on the job dependency before you schedule the first job

[hollow jolt]

ehhh... so this PolygoniserParallel is my attempt to make it multi-threaded , the one before Polygoniser is a simple IJob , but it does work on LateUpdate( )

[zenith wyvern]

Try calling dependency.Complete() BEFORE you schedule the first job

[hollow jolt]

i am rly trying to avoid that

the job runs for at least 4 frames

[zenith wyvern]

Unity needs to know the job is complete before it will let you re-use the dependency (the native container)

It's not a choice

[robust scaffold]

I do something similar here. So it's possible.

[north bay]

Accessing list.triangles.length will throw an error since the first job declares write access to the list and therefor can change the length of it on another thread

[hollow jolt]

oh well time to split the schedule and make a state machine

[robust scaffold]

Yep. The .Length is actually a redirect to .Count() which is a method and will throw the safety check.

[north bay]

You can try using AsDeferredJobArray if you don't know the final length when scheduling the read job

[zenith wyvern]

You're assigning a new native container. He's using the same one between frames

[hollow jolt]

i do this , and list . AddNoResize(new Triangle

[north bay]

AddNoResize still changes the length of the native list

It just doesn't bump the capacity when it would need to

[hollow jolt]

O.o the name doesn't fit

[robust scaffold]

You need IJobParallelForDefer. I believe it takes in a NativeList as a parameter for scheduling

[zenith wyvern]

I don't think any of that is relevant to a scheduling error is it

[robust scaffold]

it'll allow the use of the .Length of the native list following a job.

If you're using that list to iterate through the NativeList. Otherwise, I think you can just pull the .Length inside the job instead of assigning it at scheduling time.

.Count() is a readonly operation. Useable with a [ReadOnly] NativeList field of the job struct.

So just do that instead of using the .Length in the mainthread scheduling.

[hollow jolt]

i think that was it , removing the length no longer show a safety check error

im guessing accessing the NativeList<T>.length inside the Execute is very bad ?

[robust scaffold]

No? It's a method so dont use it on every index. Just use it once per index batch or the least amount of times and thats good.

[hollow jolt]

wait so when i allocate var list = NativeList<T> ( size ) and the use for( ..size )* list.AddNoResize( T ) the end result will be a list of length 2x the size ?

i think ill just stick to fixed sized NativeArray 's

[robust scaffold]

I believe, correct me if Im wrong, .AddNoResize(T) simply attempts to add a new value to a list previously allocated to capacity size and if the quantity of items in that list is already at the max capacity .Count() = size, then it would either do nothing (not adding) or throw an error.

There is no change to the length.

[north bay]

The NativeList constructor only specifies the capacity of the underlying memory not the length

[hollow jolt]

sounds like this should be faster then Add( T )

[robust scaffold]

The difference from just .Add() is that .Add will resize the buffer following the next power of 2 if the List is full but .AddNoResize() does not.

[hollow jolt]

^ this doesn't seem to work

i mean using a simple int in a parallel job

the output list does have data and length is valid

but my int counter is always at 0

( after the job is complete )

hmm tried replacing it with a NativeArray { len = 1 } and the compiler recommends using double buffering strategies

any ideas what is that ?

[coarse turtle]

double buffering is when you have a ready copy of the data you're trying to manipulate

it requires 2x memory but it allows you to have a safe copy of the data that you intend to manipulate

[hollow jolt]

what if i want a simple counter like above ?

int counter in parallel job -> Execute ( i ) => counter ++ ;

[robust scaffold]

interlocked add

or in that case, interlocked increment

[coarse turtle]

either interlock or you have per thread counters and then sum it altogether at the end of a job

[hollow jolt]

wew ... Interlocked.Increment( ref ( ( int* ) counter.GetUnsafePtr())[ 0 ] ); , instead of counter ++

it starts to feel like im doing cpp

[robust scaffold]

yep

[hollow jolt]

( panik )

[sand prawn]

Would a NativeReference make that suck less than a NativeArray[1] ?

[robust scaffold]

they are identical

[sand prawn]

Syntactically even?

[robust scaffold]

yep

[sand prawn]

So you still end up with the [ 0 ] ?

[robust scaffold]

in the case of .getunsafepointer, you still need the [0]

[sand prawn]

heh, that's funny

Looking at it, it makes sense why, but still

[hollow jolt]

well looks like it worked ( i don't understand how its possible to write to the same pointer from many treads .. intuitively there could be a collision ) /shrug

mesh.SetIndexBufferData doesn't allow me to use the counter tho ...

ArgumentException: Accessing 36120 bytes at offset 0 for mesh index buffer of size 3612 bytes is not possible.

the main idea was to make a fixed sized container that will compensate for max amount of * data for the grid size - and use the counter to know how many polygons had been created

[sand prawn]

@hollow jolt The atomic increment uses thread primitives in the processor architecture to ensure that the collisions are safe

[hollow jolt]

that's pretty nuclear

[sand prawn]

https://www.internalpointers.com/post/lock-free-multithreading-atomic-operations <--- for folks who want the propeller-head version

[amber flicker]

And usually slow. Wonder how much faster it ends up being in parallel.

[hollow jolt]

the counter itself is like 1% of the workload in the job ( or less )

[amber flicker]

But if you run the whole job single threaded, how much slower is it?

[hollow jolt]

¯_(ツ)_/¯

i still didn't port it to parallel got the SetIndexBufferData to solve

then i can compare how much performance boost it is

right now i got it in single thread burst compiled which seems super fast

but im planning to run that on the Quest so no matter how good the results are on the PC , i would need more power for the mobile chip

[amber flicker]

This post by Joachim is what made me averse to using atomics btw: https://forum.unity.com/threads/could-componentdatafromentity-t-have-a-return-by-ref-version-of-indexer-for-atomic-operations.866524/#post-5705032

[viral sonnet]

Interlocked > deferring to another job, especially for the damage to target health

tested this quite heavily. so, don't believe any theory, test it and measure it. having only a single Interlocked.Add is not only very simple but fast. If you need more than that it goes downhill

[gilded palm]

Hi, What could be wrong with ECS physics setup?

I have only 500 entites with capsule collider (kinematic)

[frosty siren]

How to pass NativeStream.ForeachCount to schedule of IJobParallelForDefer?

[amber flicker]

Sure burst enabled and safety checks disabled?

[viral sonnet]

Reading the thread that Timboc linked, I'm still amazed Unity has no real answer for such a basic game mechanic. 🤔

[amber flicker]

Is this due to job dependency overhead in this case?

[gilded palm]

oh that helps. Thank you

[viral sonnet]

it's the sum of all the overhead. allocating a container, writing to it, scheduling another job, reading. atomics may be slow but all this is slower. I don't see any real arguments unless a better practice is presented

[gilded palm]

How can I make kinematic body( with code)?

[amber flicker]

Well I could see how it’s faster in this case for now - I hope it’s not after the incoming optimisations but who knows

[viral sonnet]

the allocating and write speed is the slowest part, so scheduling optimisations won't help much

[gilded palm]

well still bad when I have 1000 entities

[amber flicker]

Hmm it sounds pretty specific- you don’t usually need to allocate a new container each run but anyway.. sleep time zzz

[gilded palm]

hi, if I have physics simulation disable (stepphysic system), do I have to call anything like Physics.SyncTransform like in Mono?

[frosty siren]

Is there a way to copy value from one NativeArray to another without knowing it's type?

[left oak]

I think you at least need to know the type size to reinterpret, but im not sure

[frosty siren]

i'm not about reinterpret, because afaik this method doesn't allocate new array it just gives you another generic wrapper to access data.
I want to copy element by element (for example from index 0 to index 99) without knowing type. I think there is some unsafe memcpy things but maybe there is also some safe extension

[robust scaffold]

Knowing the type is important to identify the stride required to allocate in memory of a new native array.

[unborn totem]

I think I figured out a way to copy component data from one entity to another with Burst compatible code thanks to the asmref trick to "extend" EntityManager and expose some of the internal methods hidden in there

big thanks to stumbling on @ocean tundra who talked about the ability to do that way back in March

I also had issues with Rider giving me phantom errors until I shut off the "Use GUID" checkbox for the asmref in the Unity inspector. Rider did not care for the GUID, it wanted the proper namespace name.

[ocean tundra]

Have there been any updates on DOTS?
The lack of visible progress and updates + life stuff made me put down my DOTS projects since then

[unborn totem]

anytime I make changes to my partial EntityManager struct, i do have to restart Unity though. Seems like something screws up during compiling and restarting the editor fixes.

there was a forum post recently about the future of DOTS

https://forum.unity.com/threads/dots-development-status-and-next-milestones-december-2021.1209727/#post-7724119

[last swan]

Can I get the current time in a bursted parallel job?
With time I mean the time since startup/beginning of the application in seconds as a float. I tried UnityEngine.Time.time and (float)World.Time.ElapsedTime, but both are only allowed on the main thread and without burst

[amber flicker]

you just have to cache it outside the job currently

[last swan]

🤦‍♀️ my big brain moment there haha thanks for the hint 🙂

[amber flicker]

ha, not at all - don't think it's that intuitive. Different issue but if you're using the foreach's sometimes you'll need to cache member fields in the OnUpdate too otherwise the safety system can't tell it's safe.

[last swan]

Ahh ok, never encountered this - thanks!
but I already cache data in the OnUpdate - and I did not think of caching the time as well, in my case it should have been intuitive enough

[pulsar jay]

Why does this happen? Ignoring invalid [UpdateAfter] attribute on Events.EventSystem targeting Unity.Entities.BeginInitializationEntityCommandBufferSystem. Is anything special about the BeginInitializationEntityCommandBufferSystem?

[UpdateAfter(typeof(BeginInitializationEntityCommandBufferSystem))]```
It is in the same group and both use OrderFirst

[devout prairie]

Probs just that OrderFirst is conflicting as there can only be one valid OrderFirst..

Or simply that it can't be first in the group but also appear after BeginInitECBSys

[pulsar jay]

yeah maybe thats it. that makes it hard to properly sort it though. How would I tell it to run after the command buffer but before everything else? 🤔

[devout prairie]

Possibly just do UpdateAfter(ecb) and if that isn't enough add an additional UpdateBefore(whatever system currently runs after the ecb)

It actually works quite well once you figure it all out

The approach i've taken is declare my own update groups like this:

So i've declared specific ECB's that each update group will use..

Then when i want a system to run inside each group, i'll also grab the ECB from the system group itself ( so any systems running in those groups always use the ECB that is assigned to that group, if that makes any sense ):

[pulsar jay]

Interesting approach 🤔 . Although I dont usually tend to use an ecb system per group. I tend to select them by usage. E.g. always destroy entites only in BeginInitialize

[devout prairie]

Basically in the code you've shown, i think all you need to do is remove the second line ( UpdateAfter ) to get it to run before that ecb )

[pulsar jay]

I also figuered it would be fine to run before that ECB because the EventSystem is basically an ECBS itself it should not really matter if it runs before or after the other ECBS

Funny thing is this gives me this warning:
Events.EventSystem has invalid UpdateInGroup[typeof(Unity.Entities.InitializationSystemGroup]

[devout prairie]

Well just be aware that you could be deferring your ecb commands until the next frame.. say if for example you have a command inside SimulationSystemGroup that uses BeginInitializationEntityCommandBuffer.. those commands wont execute until the next frame when the beginInit system runs again.. so if you want really specific order of ecb commands being consumed actually during the same frame, you have to be really specific about when and where you use the different ecb's

if that makes any sense at all 😛

[pulsar jay]

Yeah I know. But also each ECB that is actually used creates a sync point. So a try to avoid a lot of ecb systems in the middle of frames if possible

[devout prairie]

Try just:

[UpdateInGroup(typeof(InitializationSystemGroup), OrderFirst = true)]

[devout prairie]

Yeah this is true, only use if you need to

[pulsar jay]

I did. But I just figured that using [WorldSystemFilter(WorldSystemFilterFlags.All)]
might cause that warning

I guess the group does not exist at all places where the world is created

so if the world is being created without this systemgroup in any case (e.g. in editor or during tests) it throws this warning 😬

[devout prairie]

Hmm i'm not sure actually, i've never used BeginInitializationSystem ecb, maybe try EndInitSys ecb

[pulsar jay]

I dont think that will change anything. the problem is my event system. I think added the WorldSystemFilterFlags.All attribute because it needs to work during edit mode 🤔

[devout prairie]

Ahh that is something i've never tried

[pulsar jay]

Maybe I need to set more specific flags. I am not even sure what some of these mean 😅

[devout prairie]

🏁

[last swan]

https://docs.unity3d.com/Packages/com.unity.collections@0.2/api/Unity.Collections.NativeStream.html

What is "foreachCount"? I tried to use a NativeMultiHashMap but I encountered Capacity problems. I found after some research the NativeStream which could probably save my problem, but I don't really understand this value. I had a look at the Tests, and there I could understand that it is how many threads are running parallel? Or am I completely wrong 🤔

[coarse turtle]

that'd be correct, since you can use a NativeStream and each 'block' in the range can be deisgned to be used for a specific thread.

[devout prairie]

kinda hard to say without context but normal circumstances inside ForEach a native collection is 'split up' amongst parallel threads, so if your array is say 30 length, and running on 3 threads, thread 0 will have access to 0-9, thread 1 has 10-19, etc

NativeDisableParallelForRestriction gets around this at least in my case, so long as you know you are only working with unique indexs

[last swan]

Can I use NativeStreams without knowing the foreach count? esp without setting the foreach count?

[coarse turtle]

I doubt it since, it would probably not allocate anything. Internally it would do a large allocation based on the stream size * foreachcount.

[last swan]

Oki thanks 🙂

[pulsar jay]

Is there an equivalent to WithAll for ComponentTypes in CreateEntityQuery? There is an Exclude and a ReadOnly but nothing like Include?!

[north bay]

ReadOnly is the equivalent of WithAll

At least if I remember correctly

[devout prairie]

Basically everything inside it is WithAll unless you use EntityQueryDesc to specify mix of All/Any/None etc..
https://docs.unity3d.com/Packages/com.unity.entities@0.14/manual/ecs_entity_query.html

[pulsar jay]

There is a difference between WithAll and ReadOnly isnt it? Because ReadOnly requires its readable and WithAll only requires it to exist

[devout prairie]

Well adding ReadOnly flags it as being read only ie more efficient if you don't need to write to it, but with or without ReadOnly it will still be WithAll

So in the docs:

EntityQuery query
    = GetEntityQuery(typeof(RotationQuaternion),
                     ComponentType.ReadOnly<RotationSpeed>());

Both of these components are WithAll ie they are required

[pulsar jay]

I mean there is a difference between Foreach.WithAll<ComponentA> and Foreach(() => in ComponentB)

the first one will not block any jobs from writing to it for example as it does not even need to read

then again I am unsure if it does even matter in this case as I dont know if queries created from EntityManger can even be used in a parallel context

[devout prairie]

Ah right i understand.. My understanding is when using for example:
ForEach.WithAll<PlayerTag>(
Is useful because it means PlayerTag is required and only entities with PlayerTag will be included in the ForEach loop.. However the PlayerTag component will not be available for read or write within the loop..

Whereas, if you did:
ForEach(( in PlayerTag playerTag)
Would mean that PlayerTag is required, but you also intend to read data from it

[pulsar jay]

Yes exactly. And the the equivalent for in is ReadOnly. So whats the equivalent to WithAll?

[devout prairie]

ref

if you mean how do you write to it?

[pulsar jay]

no ref = ReadWrite

WithAll = dont read or write, just exist

[devout prairie]

ReadOnly is generally used for passing in for example arrays, not normally for components of the loop

[pulsar jay]

maybe it just isnt necessary as EntityManager queries cannot be used for dependency management between systems afaik 🤔

[devout prairie]

Hmm i'm not exactly sure what you mean, sorry!

Generally you create EntityQueries in OnCreate as members of the SystemBase itself, then i think when you use those queries inside OnUpdate, the system runs a job filling the query.. afaik it is possible to chain the dependency of that query into your ForEach.. but i'm really not sure if that's what you mean

[pulsar jay]

Well, nevermind. ComponentType only has these 3 AccessModes:

ReadWrite,
ReadOnly,
Exclude

So obviously there is no "Include" or "Exists" or anything similar

[devout prairie]

Hmm yeah i'm not quite clear on what you're trying to achieve as you've mentioned EntityQuery and also ForEach.. maybe if you explain what you need

Or i'm being a bit thick, which is totally possible 😛

[pulsar jay]

There is no specific need here. I was just wondering whether there is "more performant" way to create a query via EntityManager if you dont need to read or write from the specified component but just need the entities that have this component

or maybe phrased differently: Is ReadOnly the most restrictive Component type in an EntityManager query?

It looks like it is...

[devout prairie]

Ahh now i understand.. Yeah that is interesting, although thinking out loud i'd add this caveat:

The components within the EntityQuery are not in fact gathered into memory until you use myQuery.ToComponentsArray

So to specify Include wouldn't have any real purpose..

If you actually need to use the components you'd use:
myQuery.ToComponentsArray<MyComponent>();

[pulsar jay]

You could use ToEntityArray for example? That would not require access to the components

[devout prairie]

ToEntityArray basically returns an array of the Entities themselves

ToComponentsArray<MyComponent>() returns an array of the specified components

[pulsar jay]

Or in my case I use the query to add a component. Its just For all entities that have component A, add component B

I dont need to read or write to component A to do that

[devout prairie]

Indeed

[pulsar jay]

But maybe I am just overthinking. The difference between WithAll, ref and in is really important for System dependency managment. But maybe it is unimportant when used with EntityManager?

[devout prairie]

i'm not sure where EntityManager comes into this 😛

[pulsar jay]

var query = EntityManager.CreateEntityQuery(ComponentType.ReadOnly<LaneStart>());
EntityManager.AddComponent<GenerateLane>(query);

I guess in this case it does not really matter as it is executed on MainThread anyway. There is no dependency managment going on.

[devout prairie]

WithAll in and ref don't really impact dependency management either as i understand it.. you can chain any number of ForEach jobs that use totally different entity/components.. basically dependencies are for ordering the jobs not really for saying 'these jobs share the same query list'

[gilded palm]

Can i count Entities.WithAll.WithNone... without NativeCounter?

[pulsar jay]

Oh it totally does. If you use ref in a Foreach query it will ensure that Systems that run later and use in (read this component) will wait for this job to finish first

[devout prairie]

What you would do in this case is:

var query = EntityManager.CreateEntityQuery(ComponentType.ReadOnly<LaneStart>());
var entitiesFromQuery = query.ToEntityArray();
for(int i=0; i<query.CalculateEntityCount(); i++)
{
  EntityManager.AddComponent<GenerateLane>(entitiesFromQuery[i]);
}

[pulsar jay]

If you just use WithAll in the second job it does not have to wait for the first to finish as the first one will only change its contents and not its existence

[devout prairie]

Ah that's true i think yeah

I think

[pulsar jay]

So there is a big difference for ForEach. But I guess there is no need to manage dependencies with EntityManager.CreateEntityQuery so it does not have anything like WithAll

Its just a bit confusing as the CreateEntityQuery API kind of mirrors the ForEach API but not exactly 😬

[devout prairie]

it is a bit confusing in a way yeah, i think once you get comfortable with using it it becomes clearer

as you mentioned EntityManager will be on the main thread so there's no real guarentee whether ForEach jobs from Systems will be created in any useful or specific order in relation to EntityManager

[robust scaffold]

EntityManager doesnt *have* to be on the main thread. I'm doing tests now and you can use ExclusiveEntityTransaction on bursted job threads. Dont know if it works in parallel though

[viral sonnet]

ExclusiveEntityTransaction is an immediate change, right?

[robust scaffold]

Yea. It's a job accessible version of EntityManager.

[devout prairie]

Could that be used inside ForEach to SetName on Entities?

Currently i have to refactor ForEach into non parallel things to allow naming of entities for debug purposes

Be useful if ECB had SetName

[robust scaffold]

Exclusive is the backing method of EntityCommandBuffer. Anything you can do with a ECB, you can do with an Exclusive. An ECB just plays back to an exclusive.

If you cant do it with an ECB, you cant with Exclusive... well kinda. You can do ExclusiveEntityTransaction.EntityManager.SetName(entity, name). Not bursted but possible on a job thread.

[viral sonnet]

think the main point of the command buffer is that it can be used in parallel

[robust scaffold]

It's a glorified native multi hash map

[viral sonnet]

i've never used or needed ExclusiveEntityTransaction. documentation is abysmal for it

[robust scaffold]

If you look at the playback of a ECB, it's singlethreaded on the main thread. You can pass an ExclusiveEntityTransaction into a ECB to get it on a job thread but still single.

[viral sonnet]

don't get me started. ecb being single threaded is one of those things ...

[robust scaffold]

EET can not be used in the main world due to how the scheduling of systems work. Well it can be used if you're scheduling a job using it then immediately completing it within the same system OnUpdate.

[viral sonnet]

but I guess, that's not even the main problem with it. it's not really batched is more it

[robust scaffold]

ECB cant automatically detect batching. You need to do that yourself with EntityManager.DoSomething(EntityQuery, Something).

[viral sonnet]

it rather painfully creates an entity one after the other 😄

[robust scaffold]

Yep.

[viral sonnet]

it should be smart enough to detect the same archetype and group them together

especially when you create with an archetype

[robust scaffold]

Can IJobFor operate on managed arrays?

Time to see...

[viral sonnet]

i think it does when it's not burst compiled

[robust scaffold]

I know burst adds all these requirements for native containers only to pass between job and mainthread boundaries but without it... can I pass in managed stuff?

if I can, this is revolutionary

im gonna throw together some tests

because then I can fully multithread my loading

[viral sonnet]

everything goes when not being burst compiled

[robust scaffold]

I've been blinded so much by burst, I forgot I'm writing in c#

[viral sonnet]

haha, I know how you feel. Was at the same point sometimes. Like, no burst, ohhh it's so easy! I can do everything!

[robust scaffold]

When I first offloaded my loading process to an IJob, I felt so dirty passing around strings.

[robust scaffold]

InvalidOperationException: ReadCacheAuthoring.Provinces is not a value type. Job structs may not contain any reference types.

Yea, I thought so. Shame.

Cant pass class between main thread - job boundary even without burst

[viral sonnet]

maybe there's a way. Entities.ForEach certainly can

oh but that's with .Run(), guess there's the caveat

[robust scaffold]

You can call .Dispose() in a job. Well, there's no error and it seems to work.

[devout prairie]

Idea for a dots / burst GameOfLife concept:

Covid Natural Immumity vs Vaccinated within a population

Taking into account risk factors of covid vs vaccine across different age stratifications

Testing the 'vaccinate everything that moves' approach vs 'protect the vulnerable' approach

[robust scaffold]

Spatial lookups are a real pain in the ass. Look up boid optimization.

[devout prairie]

Yeah i haven't looked at the boids example in detail tbh

What's the issue vs just classic spatial lookup stuff

[robust scaffold]

That concept is basically a boid simulation. Spatial partitioning will be the primary driver of performance in that

[devout prairie]

kinda seems like btree or octree or whatever would benefit from burst no?

[robust scaffold]

There are a lot of debate on the forums for best tree organization. I still need to dig into that myself so I have no clue right now

[devout prairie]

just to qualify this, what i mean is : natural immunity should be sterilizing immunity ie people stop becoming vectors for the virus ie they can't spread it - whereas vaccinated immunity it's basically a free-roam scenario with the virus just continually spreading between people.. i'd love to test this out in a simulation

[robust scaffold]

Honestly, there's plenty of papers on that. Just check up google scholar for one.

https://www.medrxiv.org/content/medrxiv/early/2020/03/30/2020.03.25.20043109.full.pdf

[devout prairie]

Yeah i've read and listened to a lot of stuff on it, really interesting

[robust scaffold]

https://content.iospress.com/articles/clinical-hemorheology-and-microcirculation/ch209006

Covid 19 threw literal billions of dollars into the immunology field. There's papers on everything if you just look for it. Someone probably already made that simulation

[devout prairie]

I'd love to see it tbh

In scotland we have huge vaccine coverage, from age 12 up, we've did harsh lockdowns etc, but this year we have had up to 20% excess deaths over the 10 year average on a month to month basis. All i know is we've failed.

[robust scaffold]

https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2780539

[devout prairie]

Currently looks as though the measures have killed way more people than the virus itself. Which is what i predicted 20 months ago.

[robust scaffold]

Couldnt find the simulation code but the paper describing its use and results are there

Our study suggests that, for a population of 10.5 million, approximately 1.8 million infections and 8000 deaths could be prevented during 11 months with more efficacious COVID-19 vaccines, higher vaccination coverage, and maintaining NPIs, such as distancing and use of face masks. Moreover, our findings highlight the importance of continued adherence to NPIs while the population is vaccinated, particularly under scenarios of lower vaccine efficacy and coverage. Maintaining NPIs throughout the 6-month vaccine distribution period appeared to reduce infections to levels seen at the beginning of the pandemic. In contrast, under scenarios with low vaccine efficacy and coverage, premature removal of NPIs could result in a resurgence of infections with a magnitude exceeding that before vaccine distribution.

Key part:

In contrast, under scenarios with low vaccine efficacy and coverage, premature removal of NPIs could result in a resurgence of infections with a magnitude exceeding that before vaccine distribution.

Guess what happened in reality eh

[devout prairie]

And here we still are

my kids hardly seen school last year at all

[robust scaffold]

Published June 2021. Typically papers are finished about a month or two before they become public so May 2021. Just when lock downs are being lifted. Early some immunologists may say...

The statistics say that we should be locking down until 90% - 95% are immunized. Of course that is completely unrealistic.

But thats what the statistics say. These deaths are going to happen, with the lockdown or not. Locking down just stopped the first wave of deaths. The second, third, and now fourth waves are going along as nature does.

[devout prairie]

It's like trying to catch minnows with a crab net... there's evidence to suggest small measures will get you 80% of the way there but beyond that it's just all harm to society

[devout prairie]

exactly yeah

what i'm curious about is..

there's only 1 under 18 afaik in scotland that's died specifically of covid

but we've rolled out vaccines for everything above 12 yrs old ( not mandatory though )

[robust scaffold]

Lockdowns are not for the kids. Its for the people at home that the kids go back to after school

[devout prairie]

so if kids don't die.. and natural immunity is sterilizing ( ie they cannot get reinfected and continue to spread it ) then wouldn't that make more sense

[robust scaffold]

If ya want to know more, look up the literal half a million papers written about this topic. There's the answer to all your questions and more.

[devout prairie]

Yeah i mean i'm double jabbed personally, but i'm more of the opinion that really the vulnerable should be ring-fenced by the healthy, and anybody who is vulnerable should get jabbed.. but not forcing everybody to get it

just my 20 cents

i guess it takes the benefit of natural immunity and the benefit of vaccine protection

and doesn't kill 10's of thousands of people who've missed doctors appointments and not been diagnosed and suicides and breakthrough infections and all the rest

but yeah, a GameOfLife on this would be.. interesting!

Apparently Conway, the inventor of Game of Life, died of Covid 😐

[robust scaffold]

F

[visual tundra]

when considering natural immunity, also consider exposure to prior corona viruses. There's a not insignificant number of health care workers who were heavily exposed to this one, not vaccinated and never got sick. And this extends out into the normal population, too. There's been at least 7 prior corona viruses. These provide some degree of protection, according to some modelling, and this is regionally being investigated, and demonstrably shown around the south of Asia, even into Australia and NZ.

[visual tundra]

Due to the imperfection of the vaccines, and the resultant evolvement of the virus if the majority are vaccinated, thereby making the vulnerable more prone to exposure to evolvements, some consider it heinous to vaccinate anyone other than the vulnerable.

[sinful cipher]

Is there a way to know if a piece of code is executing within a (Burst) job?

[robust scaffold]

Trusty Debug.Log("Test");

Never fails

I have not touched a debugger in months. Debug.Log is all that I need.

[hollow jolt]

So i finally got my parallel polygon worker working and its 10 times slower then the single thread identical one

smh

[hollow jolt]

VS single threaded one ...

this is the only Allocation im making inside the Execute( int index ) method :

basically:

[BurstCompile(FloatPrecision.Standard, FloatMode.Fast, CompileSynchronously = true)]
public unsafe struct PolygoniserParallel : IJobParallelFor
{
  /// Persistant Containers ... 

  public void Execute(int i) 
  {
    var cell = new Cell().Allocate(Allocator.Temp);

    /// DO SOME MATH ...

    cell.Dispose(); 
  }
}

is it possible to know which core the Execute method is running on ? I'm thinking to allocate like N amount of those cell's ( if its possible to know the amount of cores that the job would run on ) and then use the Execute index to find the current active core and use the persistent container ? Or am i missing the point and the slow down isn't because I'm allocating & disposing a temp container

[dense crypt]

ForEach jobs you can pass int nativeThreadIndex as 3rd parameter

https://docs.unity3d.com/Packages/com.unity.entities@0.17/manual/ecs_entities_foreach.html#special-named-parameters

[hollow jolt]

im not using DOTS

figured out there is System.Environment.ProcessorCount so im gonna try implement a similar thing in Jobs+Burst only

unless there is already a way to access nativeThreadIndex using Jobs only ?

[last swan]

Me again.. I need some help with completeReplayHistory, I only added the relevant code and I commented my question in there

protected override void OnUpdate()
    {
      // Only run this System, when the Record Mode is set to Saving (runs only once)
      if (ReplayUtility.Mode == ReplayUtility.RecordMode.Saving)
      {
        // ToDo: test the capacity ######### problem capacity too low.
        // |key|*|value| = capacity, but |key| and |value| are unknown
        NativeMultiHashMap<int, ReplayHistory> completeReplayHistory = new NativeMultiHashMap<int, ReplayHistory>(512, Allocator.TempJob);
        
        var lookupRotationBuffer = GetBufferFromEntity<RotationDataBuffer>();
        var lookupPositionBuffer = GetBufferFromEntity<PositionDataBuffer>();

        Entities
           .WithAll<ReplayTransform>()
           .WithBurst()
           .WithNativeDisableParallelForRestriction(lookupRotationBuffer)
           .WithNativeDisableParallelForRestriction(lookupPositionBuffer)
           .WithNativeDisableParallelForRestriction(completeReplayHistory)
           .ForEach
           (
             (
               Entity entity
             ) =>
             {
        // Fill the completeReplayHistory until capacity is full - what then? or use another container? how?
        // I have an int associated with many ReplayHistory, so in general I would use: 
        // NativeHashMap<int, NativeArray<ReplayHistory>>
        // but this would be nested containers, it is not burstable and parallel anymore
             }
           ).ScheduleParallel();
        this.CompleteDependency();

        // Save the completeReplayHistory
        ReplayUtility.Mode = ReplayUtility.RecordMode.NotRecording;
        SaveLoadManager.SaveReplay(completeReplayHistory);
        completeReplayHistory.Dispose();
      }
    }

[dense crypt]

This might be what you're looking for
https://docs.unity3d.com/ScriptReference/Unity.Collections.LowLevel.Unsafe.NativeSetThreadIndexAttribute.html

[rotund token]

There's an attribute that will inject this in an int field. [native set threads index] or something, not at home to check

Beaten by peppej

[hollow jolt]

thanks , gonna try something like this

public unsafe struct PolygoniserParallel : IJobParallelFor
{
  public NativeArray<int> _data; 
  [NativeSetThreadIndexAttribute] public int threadIndex;
  public void Execute(int i)
  {
    for(var tri = 0; tri < 128; ++ tri ) 
    {
      var data = _data[ threadIndex * 128 + tri ] ;
    }
  }
}

main()
{
  new PolygoniserParallel {
    _data = new NativeArray<int> ( System.Environment.ProcessorCount * 128 , Allocator.Persistent );
  }
}

heh...

iirc there was some safety attribute to solve this if im not wrong

something like [Disable Parallel .. ?

[dense crypt]

NativeDisableParallelForRestriction
or
NativeDisableContainerSafetyRestriction

Use at your own risk

[hollow jolt]

what's the difference ?

i guess i need the first one since its a parallel job

seems that the first one worked , no more errors

[viral sonnet]

use ProfilerMarker to find out what's going on inside the burst job

and do you have to allocate in the job? looks weird to me

[visual tundra]

There's strong arguments in favour of this approach. The more I use Unity, and Rider's debugger, the more I realise this is the better approach, especially in conjunction with small, incremental development of the bigger, more complex things, etc. Thank you. It was a relief to read this.

[hollow jolt]

well its finally working but ... both the parallel job and the single threaded job have nearly identical execution times 🤔

[amber flicker]

are you still having those massive red bars where it fallsback to the other allocator? I'd avoid allocating so much within a job if you can

[hollow jolt]

so when i bump up the grid size of 120x120x120 i can finally see the difference , single thread takes about 0.2 seconds to execute , while the multi threaded code takes around 0.15 seconds on average

[hollow jolt]

no i stopped allocating anything inside the job

[hollow jolt]

^ used this way of reading / writing pre allocated memory based on the thread index

so odd, i though parallel working would give massive performance boost

[amber flicker]

I mean... I'm not going to dive into this but just to ask the obvious question but two thoughts:

1. If the job is short (<0.1ms), I would expect the overhead from schedulling to potentially mostly cancel out the gains
2. If the job is longer (your screenshots show more like 73ms), are you sure you're not e.g. accidentally doing all meshes on each thread

[hollow jolt]

pretty sure 2) i only compute the relevant polygons

it does becomes quicker when using massive grid sizes tho

0.2 s , vs , 0.15 s

here is something odd

whenever my total grid size ( X * Y * Z ) is above ushort.MaxValue value ( 65535 ) the parallel worker seems to break - doesn't seem to effect the single threaded one - the solver math is identical - also used mesh.indexFormat = UnityEngine.Rendering.IndexFormat.UInt32; in both cases

[amber flicker]

did you end up using atomics/interlock?

[hollow jolt]

yes only this one line :

public NativeArray<int> counter; /// JOB VARIABLE
...
counter = new NativeArray<int>( 1 , Allocator.Persistent ) /// MAIN THREAD 
...
Interlocked.Add( ref ( ( int* ) counter.GetUnsafePtr())[ 0 ] , 3 ); /// INSIDE JOB EXECUTE 

[amber flicker]

That's my only guess - with jobs running that long I'd expect a roughly linear speedup with number of cores

[hollow jolt]

u mean speed wise or that broken mesh anomaly beyond the ushort.max value ?

[amber flicker]

speed wise - no idea about the ushort issue

[hollow jolt]

hmmm maybe ill try adding ProfilerMarker for the actual math part to compare the two

[viral sonnet]

yeah seems like the work done doesn't justify the parallel scheduling. for anything that's <1ms I wouldn't bother with parallel processing

[hollow jolt]

any examples how to use ProfilerMarker inside a job ? I'm getting burst errors : Loading a managed string literal is not supported & A static constructor on type 'MCTerrain.PolygoniserParallel' is mixing managed and unmanaged code which is not supported. In order to solve this, please move the managed code or unmanaged code to a different class/struct ( no matter where i put the marker ) :

public static readonly Unity.Profiling.ProfilerMarker profilerMarker = new Unity.Profiling.ProfilerMarker("Math 2");

[viral sonnet]

this line belongs in the class that's scheduling the job

[hollow jolt]

so should i wrap it around the Schedule method ?

what i did is :

public void Execute(int i)
{
  profilerMarker.Begin();
  ...

[viral sonnet]

that's correct. also End() needs to be called

[amber flicker]

in case you missed it (I did), Wad1m's times are in s not ms - they're well past scheduling overhead

[viral sonnet]

ok 🙂

[hollow jolt]

I'm guessing u mean its correct to wrap it around the Schedule method , the way i got it set up , doesn't make it possible , any other way to profile inside Burst Job ?

( for reference i Schedule on LateUpdate() and check if handle.isComplete on Update() , sometimes it takes like 10 frames or more to complete if the grid size is large enough )

( and im testing this only 1 chunk so far , maybe if its quick enough ill use my schedule frame worker to chain multiple jobs without actually scheduling the handle dependencies )

[viral sonnet]

I don't know what you mean with wrapping around the schedule? Inside the execute is fine. You'll see all the profile markers in the profiler window even when there are multiples

[hollow jolt]

ah i did that but Loading a managed string literal is not supported

tried putting the static readonly ProfilerMarker inside the job / in the main thread ...

[viral sonnet]

but you get the full execute loop timing anyway so it's more interesting which parts take longer

[hollow jolt]

yes i wonder that too

[viral sonnet]

no, not inside the job. inside the class that schedules the job. it's like a singleton in your main class

then you have a ProfilerMarker in your job struct which you reference when newing the job struct

[hollow jolt]

yep tried that as well

same error

[viral sonnet]

remove the public

[hollow jolt]

sorry i might be having a smooth brain time with this simple one

inside Execute and reference is outside struct in the main thread ?

[viral sonnet]

that's inside my job

simple parameter 🙂

[hollow jolt]

ah so the static is bad

[viral sonnet]

no call to the static one

yes

here I set the job parameters

[hollow jolt]

got it , wasn't sure why remove public

[viral sonnet]

wasn't sure if that triggers your problem before you posted the code

my static profilemarkers are private

[hollow jolt]

this is very odd

[viral sonnet]

can you post the full code to smth like pastebin?

seems the work is not really split up in parallel

[hollow jolt]

yea ill make a repo in a bit , its a big chunky to make 1 file

[sinful cipher]

No I mean literally distinguishing between running in a (burst) job or running outside of a job. Not simply whether or not the code runs at all.

Say I have a method that is called both from main thread code and also from a burst job

and I wish to figure out which is the case at runtime

[hollow jolt]

void MyMethod( ...params , bool isCalledFromBurst = false ) { ... } ?

or just make a wrapper MyMethodBurst( .. params ) => MyMethod( .. params , true ) ; ?

[sinful cipher]

Nothing better? some #define or constant I can access?

@hollow jolt

[hollow jolt]

doesn't seem like there are any , i could be wrong tho : https://docs.unity3d.com/Packages/com.unity.burst@1.2/manual/index.html

[hollow jolt]

when using profilerMarkers , is this the way ?

( single threaded one , IJob ) - on the right hand side is the profiler screen shot which looks very wrong to me :/

zooming it i can see the segments , can't figure out why the large bars are there tho ...

[hollow jolt]

had another look , one of the unity devs said that its impossible - https://forum.unity.com/threads/script-define-symbol-for-burst-compile.1110704/

[sinful cipher]

Hmm, alright. Thanks

[robust scaffold]

Oh, well is you have [BurstCompile] attribute on the job struct, it will be running in burst.

[hollow jolt]

do we need to add [BurstCompile] to all static methods which are used within the job ?

[robust scaffold]

Nope. Just the header.

[devout prairie]

That's not required as such on a ForEach inside a SystemBase is it?

i mean does it try to burst it with or without that

[robust scaffold]

For each is default compiled burst

[devout prairie]

Yeah i figured it must

@hollow jolt did you get your mesh thing working, seems an interesting project was curious what exactly it was doing..

[hollow jolt]

idk yet why the 64k limit breaks the parallel job , but after a bit of refactoring its now 3x times faster then the single threaded one

( still not sure what's up with the profile markers showing up all messed up in the profile , so i just commented it out for now )

[devout prairie]

are you using some kind of meshing algorithm to build the mesh?

[hollow jolt]

ye smooth marching cubes

this one http://paulbourke.net/geometry/polygonise/

[devout prairie]

ah cool

i wonder if it'd be possible to mesh particle/cloud data from vfx graph with something like that

[hollow jolt]

never looked at vfx particle data format , i know its pretty handy for easy & neat effects

unity can bake mesh to cloud data , so there might be some source code on github , or did u mean to take existing cloud data from online scans or something like that ?

[devout prairie]

yeah just thinking out loud really as meshing can be a really cool thing to do.. was thinking say you have an interesting particle system inside vfx graph it might be possible to mesh that data.. i think vfx graph can take cloud data as an import, not sure how easy it is to get that data out though

[hollow jolt]

afaik cloud data is a "static" asset and can only be read

[devout prairie]

also curious what the 64k limit problem is about

[hollow jolt]

i bet its the parallel worker messing up the indexes

the mesh output size is beyond 64k

hmmm nvm i just tied that again and its fixed lol

refactoring solved it

so 80x80x80 grid is sampled at 100 ms in parallel , VS 150~200 ms in single thread ( that's 1/2 a million voxels O.o )

thinking to try IJobParallelForBatch ( it doesn't exists in my project so i might need to upgrade the editor version again )

[devout prairie]

I bet @robust scaffold could vectorize and squeeze the hell outta that 100ms 😛

[hollow jolt]

took a peek at the burst doc page were it says "vectorization" , don't think i will go down that rabbit hole lol , its already turbo fast as it is ( actually more then i was hoping for , originally i planned to do a frame based worker with single threaded job that makes the chunks one by one on demand , but if the ForBatch is fast enough i might just run it all at once )

[devout prairie]

nice

[hollow jolt]

ill upload it in few moment to git , its very messy but just in case ill forget about it lol

[hollow jolt]

https://github.com/nukadelic/MarchingCubesBurst

[hollow jolt]

well the 64k problem is still there ... i didn't notice the vertex count when trying larger grids

At this point single threaded and multi threaded are nearly identical in time for the grid size i need , might be just using it idk ..

[sinful cipher]

I mean for cases where a method can be called from either the main thread or a burst job, and needs to find this out at runtime.

Say if there were to be something like #if BURST_COMPILED or SomeUnityUtil.IsRunningInBurstJob, you see?

[viral sonnet]

I don't get your use case. Anway, you could ifdef the [BurstCompile] tag

[sinful cipher]

Say you have a sorting function that:

• If on main thread: starts a new job to do the sorting there.
• If already in job: simply execute right away.

Whether or not this is a good idea is not the point. The question simply is: is there a way to determine this?

[viral sonnet]

why does the method has to handle that?

in case it's a static method it can be called from anywhere, burst compiled or not

[sinful cipher]

Ok think of it this way:
You can check on what thread your function is executing with Thread.CurrentThread... So same idea, but burst-compiled jobs instead of threads.

Just say "no this doesnt exist" if it doesnt

no need to make it complicated by questioning the use case :/

[viral sonnet]

I question the use case so I can give an answer. The thing you want doesn't exist directly, that's been obvious from the answers, right?

[stiff skiff]

Just in case noone, answered. It's because your continue paths are not ending the profile marker block

I'd recommend using a using ( marker..Auto()) instead to block out areas

[molten flame]

Does DOTS animation currently support non-humanoid rigs? Struggling to get this working.

[haughty rampart]

afaik you can animate anything with the animation package. they even showed samples where they just animated a cube scaling and twisting.

[molten flame]

@haughty rampart you are correct 🙂
I got it working, I think I was just struggling with the implicit structure that is required for some animation stuff to work.

[robust scaffold]

Fuuuuck. Upgrading to 2022.1 Beta broke the graphics. Welp. Time to rename the project.

[safe lintel]

oh man animation i finally redid my ragdoll tests, I dont think the current joint conversion takes account of transform hierarchy? defaults from the ragdoll creator seem quite off for a converted entity

[robust scaffold]

Praise github. I just reverted to an old commit before I fucked around with the loading system and it works again.

[hollow jolt]

oh ... wow , smooth brain moment #3 of the week

thanks

[hollow jolt]

someone made a Job's cheat sheet : http://blog.s-schoener.com/2019-04-26-unity-job-zoo/#ijobparallelforbatch

[viral sonnet]

reminds me that I have to test out if IJobNativeMultiHashMapVisitKeyValue is faster than manual iteration in a burst job

[hollow jolt]

ayyy https://www.jacksondunstan.com/articles/7254

[gusty comet]

You mean: Praise "git". 🙂

[hollow jolt]

tried swapping IJobParallelFor to IJobParallelForBatch and the previous schedule job seem to break because of that ( its generated values are different )

the second job * even uses the same method regardless if its For or ForBatch

this is how im scheduling it :

var handle = J1.ScheduleBatch( parameters.length, 256, dependency );

///< Compared to this one : >

var handle = J1.Schedule( parameters.length, 256, dependency );

that's basically the only difference between the two

[devout prairie]

you could be right there although i haven't noticed anything too out of kilter for my purposes, essentially just set it up and left it as is

[devout prairie]

just to add fyi as i know we've both attempted this.. i've ended making my ragdoll solution one-way ie no lerping back to animated characters, i did have this working previously but for now have decided to just drop it - basically the characters ragdoll when dead and that's that 😛

currently have essentially a timer on ragdolls that destroys all joints/colliders after a certain time, just to reduce overall load when more and more mount up.. what i'd like to add is maybe instantiate a simple cylinder or box collider in place when this happens, so that at least ragdolls can still 'pile up' on top even after they've been destroyed

[hollow jolt]

can we use ref parameter to set the local var as a ref as well ? i got something like this , not sure if its the right approach

[devout prairie]

I've never did this before with ecs but when i have a gameobject in a scene with ConvertAndInjectGameObject, the entity gets a Transform component with a ref to the gameobjects Transform -
Should changes in the entity's rotation/translation be automatically propagated to the gameobject transform?
I remember hearing something about companion gameobject not sure if that's a separate thing..

[left oak]

Do you mean the CopyTransformToGameObject component?

[safe lintel]

@devout prairie I had to redo the joints for every limb, I recall the physx ragdoll maker being decent at getting defaults with regular gameobjects but the resulting converted ragdoll had terrible limits all over the place

[devout prairie]

well no there's just a Transform component on the entity..

[devout prairie]

yeah i'd like to play around with that further, limits etc too but for now, it works so i'm not touching the setup 😛

[left oak]

"Should changes in the entity's rotation/translation be automatically propagated to the gameobject transform?" If you want this to happen, I believe that you need to add a CopyTransformToGameObject component during conversion

[safe lintel]

i did notice if the char wasnt animated and in tpose, the ragdoll seemed fine? but animate then ragdoll, either explode or just incorrect limbs, prior to redoing them 🙂 it would be nice to get some answers from the anim team on that thread

[devout prairie]

thanks yeah i read about that, wasn't sure if it was outdated or not, thanks.. i could just as easily update these manually but was just curious if there was an embedded way or if it was part of the CreateAndInjectGameobject thing

[devout prairie]

the way i've did it is i instantiate the ragdoll prefab in an early ECB, then pos/rot all the ragdoll bones to the animated char bones, and so on.. as opposed to actually building the ragdoll in position..

so i have a prefab ragdoll that's spawned in which is setup and matches my characters tpose

i def had issues with things popping around etc and had to spend some time splitting the process of instantiating, positioning, and updating the animated characters AnimatedLocalToRoot etc across i think 3 separate ECB's within the frame

[safe lintel]

ah, I reworked mine so its directly on the bones, so it either follows in kinematic or is dynamic when toggled

[devout prairie]

also it seems finicky as hell as to when pos/rot'ing those ragdoll bones actually works, ie before or after the fixed physics update etc, i can't remember exactly what i ended up with but yeah that part was irritating

[devout prairie]

nice i'm curious what the differences of the two approaches are

ah so you've literally applied the joints to the animated character and toggle them between kinematic/dynamic?

[safe lintel]

yeah, you have a tons less overhead if you instantiate it later, i think i can have 3x the characters if there isnt an entire rigidbody ragdoll structure following or driving the skinned mesh

[devout prairie]

ah really, that's definitely something to think about then.. is switching to dynamic seamless and then when you switch back to kinematic are you lerping back to the animated positions?

[safe lintel]

havent yet attempted lerping to kinematic

but going to dynamic is pretty good so far?

[devout prairie]

bearing in mind i do only instantiate the ragdoll when the unit dies, so only have the overhead while it's actually active ragdoll and then don't lerp back

[safe lintel]

https://gfycat.com/violetclassicarcticduck

[devout prairie]

yeah i don't see any reason why lerping back would be a problem

[devout prairie]

looks great, the limits look nice also

so when mine ragdoll, as i said they are doing it on death - so there's generally always a force applied, either explosion or bullet impact.. basically at the moment i'm just applying this to the ragdoll chest bone.. there is occasionally stretching when joints go out of bounds of their limits, which shouldn't obviously happen, but this was kindof an issue with traditional unity ragdolls also

[safe lintel]

yeah ive yet to integrate it into projectiles and damage, hoping it doesnt go back to exploding. also wondering if im just wasting time if unity decides to release significant animation changes if that update arrives soon

[devout prairie]

I'd guess there'll be some changes but tbh, happy to roll with them, if there's an easier way of doing this in the near future i'd be glad of it

[devout prairie]

thing that annoys me is, with just the player, and one enemy, i don't hit 30fps in the editor

do you find that with urp orr?

this is with the player and one enemy

fixed step again taking up a ton of time, and culling

with max 30fps i'm only really left with around 10ms headroom for testing spawns etc before it starts to get unplayable

[safe lintel]

my culling doesnt appear to change adding more animated characters on screen

[devout prairie]

possibly the trees on my terrain that's causing the overhead tbf

yeah i do shave 10ms off that render/culling phase without the terrain/trees

i had Application.targetFrameRate = 30; for some reason

still not entirely sure what that does, or why i had it!

seems to cap the framerate in the editor tho

[safe lintel]

did that solve it?>

[devout prairie]

well removing the terrain/trees clawed back 10ms and removing the targetFrameRate got me up to 40-50fps in the editor

[safe lintel]

nice

found(after forgetting about it) theres DFG collision and raycast nodes in the physics package, wondering if that is their plan for ragdolls? not seeing any way of disabling animation currently

[devout prairie]

i think the target fps setting is forcing physics step stuff to 'fill the gap' or something, not exactly sure

[devout prairie]

dfg collision and raycast nodes?

[safe lintel]

com.unity.physics@0.6.0-preview.3\Unity.Physics\DFG\ColliderCastNode.cs & RaycastNode.cs

[runic pivot]

Hello, not sure if this is the right place to ask but I am trying to find a way to switch child gameobjects using prefabs in a Unity Tiny project. I can't seem to find any examples of this

[devout prairie]

If i have a subscene with some entities being created inside it - how do i access those entities?

Doing GetSingletonEntity<myComponent>() from one of my systems throws an error even though i know this entity/component does exist in the subscene?

[safe lintel]

it shouldnt make a difference where those entities are

[devout prairie]

yeah i'm wondering are subscenes loaded after other system instantiation or, something

i can literally see the entity exists with the component, but my system is throwing an error trying to access it, saying zero exist

first time using subscenes at all 😐

[haughty rampart]

Not sure what exactly you're doing, but subscenes are pretty much self contained. You cannot cross access entities from one subscene in another subscene, well at least not with index and stuff like you normally would. There is a way to cross access entities but I don't remember from the top of my head. @devout prairie

[devout prairie]

Hmm yeah i feel like i'm missing something obvious here..

So i have my scene, containing the usual gameobjects and some that have ConvertToEntity enabled etc.. and also a subscene with a single object which uses a Mono/IConvertGameObjectToEntity to declare some prefabs..

[robust scaffold]

Subscenes arent independent worlds though. All entities created in a standard DOTS project (not netcode) are pooled in one giant entity world. The default one. So calling GetSingleton<>() from one subscene should be able to get a singleton instantated from another.