Passing a List of DynamicTypeHandles into Job | Official Unity Discord | Page 1

limpid vault Mar 17, 2023, 2:20 PM

#

I am trying to pass a UnsafeList of DynamicTypehandles into a Job like this:

[BurstCompile]
    public unsafe struct TestDecisionJob : IJobChunk
    {
        public UnsafeList<DynamicComponentTypeHandle> considerationTypeHandles;

        public void Execute(in ArchetypeChunk chunk, int unfilteredChunkIndex, bool useEnabledMask, in v128 chunkEnabledMask)
        {
            for (int i = 0; i < considerationTypeHandles.Length; i++)
            {
                var considerationHandle = considerationTypeHandles[i];
                var considerationChunkData = chunk.GetDynamicComponentDataArrayReinterpret<LocalToWorld>(ref considerationHandle, sizeof(LocalToWorld));

                var consideration =  considerationChunkData[i];
            //Some Logic
            }
        }
    }```

#

The error i am getting : ```System::InvalidOperationException: The Unity.Entities.DynamicComponentTypeHandle has been declared as [WriteOnly] in the job, but you are reading from it.
This Exception was thrown from a job compiled with Burst, which has limited exception support.
#3 scripting_raise_exception(ScriptingExceptionPtr)
#4 AtomicSafetyHandle_CUSTOM_CheckWriteAndThrowNoEarlyOut_Injected(AtomicSafetyHandle const&)
#5 Unity.Entities.ArchetypeChunk.GetDynamicComponentDataArrayReinterpret<Unity.Transforms.LocalToWorld>(Unity.Entities.ArchetypeChunk* this, ref Unity.Entities.DynamicComponentTypeHandle typeHandle, int expectedTypeSize) -> Unity.Collections.NativeArray1<Unity.Transforms.LocalToWorld>_b8582670f79269075d32f191bad67123 from UtilityAI, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null #6 Unity.Entities.JobChunkExtensions.JobChunkProducer1<UtilityAI.Core.TestDecisionJob>.ExecuteInternal(ref Unity.Entities.JobChunkExtensions.JobChunkWrapper`1<UtilityAI.Core.TestDecisionJob> jobWrapper, System.IntPtr bufferRangePatchData, ref Unity.Jobs.LowLevel.Unsafe.JobRanges ranges, int jobIndex) -> void_b8582670f79269075d32f191bad67123 from Unity.Entities, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null
#7 d7e4b3b4cd88a8cec82a68f7ff8903d9
#8 ExecuteJob(ManagedJobData*, void ()(void, void*, void*, void*, int), int, unsigned char*)
#9 ExecuteJobCopyData(ManagedJobData*, void ()(void, void*, void*, void*, int), int)
#10 ForwardJobForEachToManaged(ManagedJobData*, unsigned int)
#11 ujob_execute_job(ujob_control_t*, ujob_lane_t*, ujob_job_t*, ujob_handle_t, int)
#12 lane_guts(ujob_control_t*, ujob_lane_t*, int, int)
#13 worker_thread_routine(void*)
#14 Thread::RunThreadWrapper(void*)
#15 _pthread_start
#16 thread_start

#

how can i declare the DynamicComponentTypehandles inside the UnsafeList as ReadWrite or Readonly?

limpid vault Mar 17, 2023, 3:08 PM

#

If this would work i could replace all my generic SystemBases with codegenerated bursted ISystems and use the default World initialization for them.

polar pelican Mar 17, 2023, 4:47 PM

#

They're native containers so can't be stored in other containers

#

https://gitlab.com/tertle/com.bovinelabs.core/-/blob/master/BovineLabs.Core/Extensions/DynamicComponentTypeHandleExtensions.cs

#

I remember having a similar issue so I wrote a story hacky method that would let me just use 1 handle which I simply changed type

#

You need to make sure you manually add all your other dependencies to the system though

limpid vault Mar 17, 2023, 5:03 PM

#

oh that sounds like a fun solution 😄
I thought unsafelist bypasses that nested NativeContainer issue. Guess i need to just codegen every single handle.
I remember someone writing something about Hybridrenderer using a struct with 128 DynamicCompTypes and passing that into the job. could I "iterate" that struct with offsets?

polar pelican Mar 17, 2023, 6:10 PM

#

Totally

limpid vault Mar 18, 2023, 5:50 AM

#

    {
        public DynamicComponentTypeHandle handle0;
        public DynamicComponentTypeHandle handle1;
        public DynamicComponentTypeHandle handle2;
        public DynamicComponentTypeHandle handle3;
        public DynamicComponentTypeHandle handle4;
    }```

#

public unsafe struct TestJob : IJobChunk
    {
        public DynamicTypeHandleContainer container;

        public void Execute(in ArchetypeChunk chunk, int unfilteredChunkIndex, bool useEnabledMask, in v128 chunkEnabledMask)
        {
            var iterations = 5;
            fixed (DynamicTypeHandleContainer* p = &container)
            {
                for (int i = 0; i < iterations; i++)
                {
                    DynamicComponentTypeHandle* iterationP = (DynamicComponentTypeHandle*)p;
                    iterationP++;
                    var dynamicHandle = *iterationP;
                    
                    //do stuff with handle
                }
            }
        }
    }

#

does that seem about right? first time ever doing anything with pointers lol

#

oh woops i got an off by one error here. but aside from that is that how i can walk through the members? just incrementing the pointer?

#

or is this only working under the assumption that there is no padding

polar pelican Mar 18, 2023, 6:07 AM

#

just treat it like an array, no need to overcomplicate it

#

DynamicComponentTypeHandle iterationP = p[i];

#

if you stored nothing but DynamicComponentTypeHandle in oyur struct you could dynamically know what your iteration count was

limpid vault Mar 18, 2023, 6:10 AM

#

polar pelican DynamicComponentTypeHandle iterationP = p[i];

i knew that was somehow possible but didnt get it to work yet.

#

not sure how i need to cast the container then

polar pelican Mar 18, 2023, 6:11 AM

#

why are you casting?

#

you already have your dynamiccomponenttypehandle just using p[i]

#

why do you need the ptr of it?

#

            fixed (DynamicTypeHandleContainer* p = &container)
            {
                for (int i = 0; i < iterations; i++)
                {
                    DynamicComponentTypeHandle dynamicHandle  = p[i];
                    //do stuff with handle
                }
            }```

#

OH

limpid vault Mar 18, 2023, 6:13 AM

#

polar pelican Mar 18, 2023, 6:14 AM

#

fixed (DynamicTypeHandleContainer* container = &Container)
{
      var p = (DynamicComponentTypeHandle*)container;
      for (int i = 0; i < iterations; i++)
      {
           DynamicComponentTypeHandle dynamicHandle  = p[i];
           //do stuff with handle
          }
}```

#

oh my naming is off

#

you get the idea though

#

i used container twice

#

if you want to make it cleaner

#

            var p = (DynamicComponentTypeHandle*)UnsafeUtility.AddressOf(ref container);
            for (int i = 0; i < iterations; i++)
            {
                DynamicComponentTypeHandle dynamicHandle = p[i];
                //do stuff with handle
            }```

limpid vault Mar 18, 2023, 6:17 AM

#

so i can skip the fixed part entirely? thats nice

#

what did you mean with i could know the amount of iterations dynamically?
My problem would be that some of those handles would go unused

#

id need to check for null each iteration to stop wouldnt i?

polar pelican Mar 18, 2023, 6:21 AM

#

you just have a hard coded 5 in there

#

you could just do sizeof(DynamicTypeHandleContainer) / sizeof(DynamicComponentTypeHandle)

#

to know how many are actually in the struct

#

if you were dynamically generating them

#

if they're invalid you're going to get safety errors on scheduling job btw

#

(i think)

#

will probably have to attribute them to turn off safety

limpid vault Mar 18, 2023, 6:24 AM

#

wait i dont exactly follow. my plan was to have a DynamicHandleContainer with 20 handles. my systems just use how ever many they would need and leave the rest empty. The systems know the count.

#

what do you mean with dynamically generating them? codegen?

polar pelican Mar 18, 2023, 6:25 AM

#

well i'm telling you that

#

if you only assign 7 out of the 20

#

safety system is going to complain you haven't created the native containers

#

unless you turn off safety on them

limpid vault Mar 18, 2023, 6:25 AM

#

hmm it somehow doesnt actually detect that. i already tried

polar pelican Mar 18, 2023, 6:26 AM

#

interesting

#

most containers fail this

#

well that's good for you

limpid vault Mar 18, 2023, 6:26 AM

#

yep lets not tell anybody lol

#

do you think it would be better to check if the handle is null and stop iterating or pass in the number of handles used into the job?

polar pelican Mar 18, 2023, 6:28 AM

#

i just thought if you were code-genning you could just code gen exact amount you needed

#

but it really doesn't matter

limpid vault Mar 18, 2023, 6:29 AM

#

this whole ordeal is to do the minimum amount of codegen necassary. i hate it 😄

#

follow up question for my own understanding:
is incrementing the typed pointer / accessing the struct as an array already compensating for padding or am i just lucky again that there is none

polar pelican Mar 18, 2023, 6:54 AM

#

Padding is included in the size of a component

limpid vault Mar 18, 2023, 7:34 AM

#

to actually use DynamicHandles i do:
var chunkData = chunk.GetDynamicComponentDataArrayReinterpret<LocalToWorld>(ref dynamicHandle, sizeof(LocalToWorld));

how do i have to reinterpret it when its a DynamicBuffer<LinkedEntityGroup> instead of LocalToWorld for example? Or do i have to use other methods to get my DynamicBuffer data

polar pelican Mar 18, 2023, 7:35 AM

#

you interpret it as byte

#

and do all you rowrk as abyte 😄

#

oh you mean as a buffer?

#

chunk.GetDynamicComponentDataArrayReinterpret<LocalToWorld>
if you're doing this seems like you could just use a component handle?

#

anyway for a buffer there is a getbufferacecssor

#

or something like that

limpid vault Mar 18, 2023, 7:37 AM

#

ah! GetUntypedBufferAccessor

limpid vault Mar 18, 2023, 7:39 AM

#

polar pelican > chunk.GetDynamicComponentDataArrayReinterpret<LocalToWorld> if you're doing th...

its just for my Test System where i have no codegen setup yet etc.
I know that every Type comming in here actually looks exactly the same. so i just have a struct thats called "GenericAxis" which i can reinterpret any type to.

polar pelican Mar 18, 2023, 7:40 AM

#

Oh interesting 👍

limpid vault Mar 18, 2023, 7:48 AM

#

oh boy im not sure if i can figure out how to do all that unsafe stuff without any materials showing some uses online.

#

can i somehow reinterpret that unsafebuffer back to a safe buffer?

#

as i know its a DynamicBuffer<GenericAxis>

polar pelican Mar 18, 2023, 7:50 AM

#

ah yes

#

buffer is a bit more complicated

#

https://github.com/tertle/com.bovinelabs.saving.free/blob/master/BovineLabs.Saving/Save/BufferElementDataSave.cs

#

i use it here if you want to see it in action

#

but this case is more of a copy paste

#

oh actually i have another use case in my core library

#

https://gitlab.com/tertle/com.bovinelabs.core/-/blob/master/BovineLabs.Core/States/StateFlagModelWithHistory.cs

GitLab

BovineLabs.Core/States/StateFlagModelWithHistory.cs · master · Tim ...

GitLab.com

#

but i actually wrote a custom

#

UntypedDynamicBuffer

#

struct to help with this

limpid vault Mar 18, 2023, 7:52 AM

#

var buffer =(DynamicBuffer<GenericAxis>*) bufferAccessor.GetUnsafeReadOnlyPtrAndLength(entityInChunkIndex,out var length);

#

might that be all thats needed?

polar pelican Mar 18, 2023, 7:53 AM

#

for yoru case maybe

limpid vault Mar 18, 2023, 8:01 AM

#

polar pelican UntypedDynamicBuffer

this is probably where GetUntypedBuffer(i) is comming from?

polar pelican Mar 18, 2023, 8:02 AM

#

i wrote it custom

#

https://gitlab.com/tertle/com.bovinelabs.core/-/blob/master/BovineLabs.Core/Collections/UntypedDynamicBuffer.cs

#

indexing just returns void*

#

instead of the element

limpid vault Mar 18, 2023, 9:12 AM

#

okay after fixing some other issue im back to this.

    {
        public int iterations;
        public uint globalSysVersion;
        public DynamicTypeHandleContainer container;

        public void Execute(in ArchetypeChunk chunk, int unfilteredChunkIndex, bool useEnabledMask,
            in v128 chunkEnabledMask)
        {
            var p = (DynamicComponentTypeHandle*)UnsafeUtility.AddressOf(ref container);
            for (int i = 0; i < iterations; i++)
            {
                DynamicComponentTypeHandle dynamicHandle = p[i];

                var bufferAccessor = chunk.GetUntypedBufferAccessor(ref dynamicHandle);

                var enumerator = new ChunkEntityEnumerator(useEnabledMask, chunkEnabledMask, chunk.Count);
                while (enumerator.NextEntityIndex(out var entityInChunkIndex))
                {
                    var buffer = *(DynamicBuffer<Consideration<GenericAxis>>*)bufferAccessor.GetUnsafeReadOnlyPtr(entityInChunkIndex);
                    Debug.Log($"{buffer.Length}");
                }
            }
        }
    }```

#

the debug log throws an error: NullReferenceException: Object reference not set to an instance of an object Unity.Collections.LowLevel.Unsafe.AtomicSafetyHandle.CheckReadAndThrow (Unity.Collections.LowLevel.Unsafe.AtomicSafetyHandle handle) (at /Users/bokken/build/output/unity/unity/Runtime/Export/Jobs/AtomicSafetyHandle.bindings.cs:165) Unity.Entities.DynamicBuffer`1[T].CheckReadAccess () (at ./Library/PackageCache/com.unity.entities@1.0.0-pre.15/Unity.Entities/Iterators/DynamicBuffer.cs:149) Unity.Entities.DynamicBuffer`1[T].get_Length () (at ./Library/PackageCache/com.unity.entities@1.0.0-pre.15/Unity.Entities/Iterators/DynamicBuffer.cs:79) UtilityAI.Core.TestDecisionJob2.Execute (Unity.Entities.ArchetypeChunk& chunk, System.Int32 unfilteredChunkIndex, System.Boolean useEnabledMask, Unity.Burst.Intrinsics.v128& chunkEnabledMask) (at ./Packages/hiautilityai/UtilityAI/Core/GeneratedDecisionSys.cs:108)

#

so it seems like my cast to a known dynamicBuffer does not work

polar pelican Mar 18, 2023, 9:15 AM

#

oh yeah that doesn't work

limpid vault Mar 18, 2023, 9:16 AM

#

#

this is how a Typed DynamicBuffer is returned by BufferAccessor

#

GetUnsafeReadOnlyPtr should return what is "hdr" in the screenshot right?

#

do i just have to create a new Dynamicbuffer too?

polar pelican Mar 18, 2023, 9:17 AM

#

they don't have the same layout

#

#

so can't cast them

limpid vault Mar 18, 2023, 9:22 AM

#

hm i dont quite understand. i thought im not casting untypedAccessor to Accessor but the returned DynamicbufferHeaderPointer to a DynamicBuffer

polar pelican Mar 18, 2023, 9:24 AM

#

yeah but a dynamic buffer is much more than just the header

#

you're missing all this

limpid vault Mar 18, 2023, 9:25 AM

#

oh this is me not knowing anything about pointers. i was assuming if i get the header with a pointer and cast that to the DynamicBuffer all the fields are available to me because they would be accessed by offset from the header pointer

polar pelican Mar 18, 2023, 9:28 AM

#

it's not how they're stored unfortunately

limpid vault Mar 18, 2023, 10:01 AM

#

alright. it was way easier than i expected actually:

public unsafe struct TestJob : IJobChunk
    {
        public int iterations;
        public DynamicTypeHandleContainer container;

        public void Execute(in ArchetypeChunk chunk, int unfilteredChunkIndex, bool useEnabledMask,
            in v128 chunkEnabledMask)
        {
            var p = (DynamicComponentTypeHandle*)UnsafeUtility.AddressOf(ref container);
            for (int i = 0; i < iterations; i++)
            {
                DynamicComponentTypeHandle dynamicHandle = p[i];

                var bufferAccessor = chunk.GetUntypedBufferAccessor(ref dynamicHandle);

                var enumerator = new ChunkEntityEnumerator(useEnabledMask, chunkEnabledMask, chunk.Count);
                while (enumerator.NextEntityIndex(out var entityInChunkIndex))
                {
                    var firstBufferElementPtr = (Consideration<GenericAxis>*) bufferAccessor.GetUnsafeReadOnlyPtrAndLength(entityInChunkIndex, out var bufferLenght);
                    for (int j = 0; j < bufferLenght; j++)
                    {
                        var element = firstBufferElementPtr[j];
                        Debug.Log($"{element.score}");
                    }
                }
            }
        }
    }```

#

i missed that GetUnsafeReadOnlyPtr does not return the header of the accessor but the first element in the buffer already

#

id consider that solved for now but am sure i have follow up questions 🙂
❤️ @polar pelican

unreal plume Mar 18, 2023, 1:17 PM

#

This has been a very informative thread. Thank you both 😊

limpid vault Mar 18, 2023, 2:03 PM

#

glad that me feeling stupid helps others 🙂

wet quest Mar 18, 2023, 2:42 PM

#

I actually had a need for something like this too, my solution was to make a custom container that stored the type handles. https://github.com/JonasDeM/QuickSave/blob/master/QuickSave.Containers/TypeHandleArray.cs Maybe it is useful still.

limpid vault Mar 18, 2023, 2:52 PM

#

wet quest I actually had a need for something like this too, my solution was to make a cus...

oh wow thats so much better than my hack.
i dont understand enough of the internals of the safety system. why is it allowed to store nativecontainers inside nativecontainers in your case? is that just something each of the unity nativecontainers disallow specifically?

#

and why couldnt i then pass an UnsafeList with DynamicHandles where safety should be disregarded?

wet quest Mar 18, 2023, 4:23 PM

#

Ah yes, it only works if you disable safety on the DynamicTypeHandle. Unity modifies the safety handles via reflection on containers when you schedule a job. So for nested containers it can't do that & you get that weird error.

I'm curious actually, how does your code work? When I tried a similar approach it complained that every NativeContainer passed into the job needed to be a valid container. Or do you just assign a unique valid DynamicComponentTypeHandle to all fields?
I kinda prefer your approach if you found a way to get around this 'Valid Container' issue.

limpid vault Mar 19, 2023, 7:12 AM

#

Its just a struct like this:

    {
        public DynamicComponentTypeHandle handle0;
        public DynamicComponentTypeHandle handle1;
        public DynamicComponentTypeHandle handle2;
        public DynamicComponentTypeHandle handle3;
        public DynamicComponentTypeHandle handle4;
    }``` 

And i fill it like that: 
```cs
[BurstCompile]
        public unsafe void OnUpdate(ref SystemState state)
        {
            DynamicTypeHandleContainer dynamicHandlesContainer = new DynamicTypeHandleContainer();
            var p = (DynamicComponentTypeHandle*)UnsafeUtility.AddressOf(ref dynamicHandlesContainer);
            for (int i = 0; i < dynamicHandles.Length; i++)
            {
                var dynamicHandle = dynamicHandles[i];
                dynamicHandle.Update(ref state);
                p[i] = dynamicHandle;
            }

            state.Dependency = new TestJob
            {
                iterations = dynamicHandles.Length,
                dynamicHandlesContainer = dynamicHandlesContainer,
            }.ScheduleParallel(query,state.Dependency);
        }```

#

dynamicHandles in this case is just a NativeList<DynamicComponentTypeHandle>. I dont get any errors if some of those handles in the struct are not assigned.

polar pelican Mar 19, 2023, 7:18 AM

#

yeah i found that weird as well 🤷‍♂️

#

you definitely have jobs debugger on right

limpid vault Mar 19, 2023, 7:20 AM

#

you mean jobs -> burst -> safety checks or is there somewhere else i should look too?

#

would be weird if i had it off since the whole point was to get rid of the "Declared as WriteOnly Error" which should come from the jobs debugger

#

ah nvm

#

ever since they moved the menus im just confused 😄

#

i mean we could risk it and ask some Unity guys if thats supposed to be possible? i just need a working solution afterwards if they bugfix that xD

#

Why is it a safety issue in the first place if you pass an empty value? shouldnt this just be a warning that you might have forgotten something?

polar pelican Mar 19, 2023, 7:42 AM

#

it's just always been an error on native containers

#

            private NativeList<byte> compressed;```

#

i just have a bunch of these around

#

so yeah if it works for you all good!

#

if it ever doesn't just attribute them up

#Passing a List of DynamicTypeHandles into Job