Populate Native container from several bursted jobs

[indigo zenith]

Hi devs, a newbie need pieces of advice on the topics :

Here is the context : I'm trying to build a custom drawing gizmos system for ecs using burst and jobs.
To display the "lines" or "shapes" (meshes) I chose to use Graphics.RenderMeshIndirect in addition with a custom URP Shadergraph Shader for GPU Instancing, easy to use, fast to render for debugging purposes.
I have to feed Graphics.RenderMeshIndirect with a ComputeShader of custom draw data (position, length, width, color...) among others parameters.

Here is the expected project : I want to have the ability to call a static method (ex : Gizmos.DrawLine(DrawData data)) in any (Bursted) System like Debug.DrawLine(). This method would feed (.add) a temp NativeList<> store in the static class Gizmo. I don't know if this approach is good enough but it actually works for my project.

Here is the question : Now the problem is (bursted) jobs, I don't really know how to deal with concurrent writing. My idea is to build a custom container like an ECB with Draw() equivalent methods to pass to the job and when it is completed, use an .addRange equivalent to feed the main NativeList stored in the static class Gizmos, but right now I'm struggling with race conditions and job dependency issues.

I would be grateful to hear your thoughts on the project, potential solutions and critics, thanks in advance

[clever yoke]

This is also what I am currently working through with my own debugging solution

I also haven't figured out exactly what I'm doing yet

I also have to figure out how to detect whether something is in a fixed update context or not (from a job perspective)

[light vessel]

so a few things
just to be clear, you're using sharedstatic for this? any other approach is not going to work

native containers are never going to work either, they will always fail on safety

you need to use unsafe containers or your own solution

[clever yoke]

You can't put a native container inside of a SharedStatic anyway, so you're forced into Unsafe containers once you're forced into that choice 😄

[light vessel]

you really have 4 options

1. create a very complicated locking mechanic like the Unity.Logging package does
2. use a parallel container that you can make parallel safe with interlocks
3. do 2 but write it in native code
4. just use the existing unity physics implementation of this and saveyourself the pain?

[indigo zenith]

Well with my limited knowledge, I'm not going to recreate a Unity.Logging system. Unity Physics can draw gizmos ?
I'm gonna find documentation on interlocks, SharedStatic and native code to see if want to adventure myself in this.
Well thx for the answers.

[light vessel]

PhysicsDebugDisplaySystem

can just draw all this

you can look at how they do it

as inspiration

[indigo zenith]

Oh well, for sure

[stuck hemlock]

Can we use it or need to look at code and recreate?

[clever yoke]

You can use it

[light vessel]

Unity packages are released under Unity Companion License

which is basically, you can use it however you want as long as it's used within Unity

i.e. you can't just take the code and port it to unreal, but if you're using it in a unity project do as you want

(but read the license yourself, don't take legal advise from stranger on internet)

1. Unity Companion Use. Exercise of the license granted herein is permitted as long as it is in connection with the authoring and/or distribution of applications, software, or other content under a valid Unity content authoring and rendering engine software license (“Engine License”). That means, for example, as long as you authored content using the Work under an Engine License, you may distribute the Work in connection with that content as you see fit under this License. No other exercise of the license granted herein is permitted, and in no event may the Work be used for competitive analysis or to develop a competing product or service.

• 9 other clauses

[stuck hemlock]

Use on unity itself

[clever yoke]

I was interpreting the question to kinda be "are the functions public, can they be called; or do I need to recreate it", so the answer to that is yes

[stuck hemlock]

My game is unity based

Thank you, i trought it was closed because tertle said to look at source code and use it as inspiration

[sharp gazelle]

I can describe bone visualization system in Rukhanka that I have implemented. It is effective and whole drawing for arbitrary number of bones and animated entities is done via two Graphics.DrawMeshInstancedProcedural (one for lines primitives, and one for triangles). In a separate job I am iterating over all bones and fill special GPU bone structure using NativeList.ParallelWriter. Then, in draw call, I use bone data structure per-instance to expand and orient generic bone mesh to proper world pose. For generic gizmos draw code some sort of command buffer with parallel writing capability can be used, and then rendering process would be the same.

[indigo zenith]

Ok so the bones are entities and you iterating over in a single schedule parallel job to get drawing data.
well I think I will limit my expectation and think of a system built on entities (easier parallel create/query functions) instead of a less specific case where I need to build an entire concurrent writing system.

Thanks for your answer Rukhanka

[sharp gazelle]

Bones are not entities. This is internal Rukhanka array.

[indigo zenith]

Owh I see, false assumption. Is this array fed in parallel ?

[sharp gazelle]

Yes. All bone processing is parallel

[indigo zenith]

Do you mind if I ask what kind of struct type did you use for your array ? Like a native parallel container with interlock as Tertle mentioned earlier.

[sharp gazelle]

I am using ordinary array, that filled in jobs with [NativeDisableContainerSafetyRestriction] attribute. Before parallel processing I am calculating correct offsets for each worker.

[indigo zenith]

Oh well, pretty straight forward, I'm gonna try to use entities and your solution and compare both for my use case

[indigo zenith]

@sharp gazelle Hey I have another question if you don't mind, Are you using GPU Instancing with URP ? If so, do you use the hlsl injection hack in your shadergraph shader to make it work ? Because I have one error with this method I don't understand.

[sharp gazelle]

Hi. No, I am using pure HLSL to make my bonerenderer shaders. And I saw you question in shader section. I think your unity_LocalToWorld missing matrix is from variant with INSTANCING_ON defined (not PROCEDURAL_INSTANCING_ON) in this case there is no unity_LocalToWorld and unity_WorldToLocal matrices variables existed. They are replaced with unity_LocalToWorldArray with array access. You can try to replace unity_LocalToWorld with UNITY_MATRIX_M (which is define)

[indigo zenith]

thx for this answer I'm gonna try that to modify it, I'm a little bit frustrated because, I wrote exactly the same code as 3 or 4 implementation on that and, well it seems I am the only one getting this error ^^'

I also need to learn more about shader, it's pretty confuse right now. Do you have any learning resource ?

@sharp gazelle well, sorry to bother you but, I read in documentation (Unity 2022.3) that creating shaders that support GPU instancing is not compatible with URP, unless with the previously mentioned hack. So I left the idea to make pure HLSL, so it is possible to create a shader urp compatible with gpu instancing ?

[sharp gazelle]

Sure thing. URP and HDRP is just a renderer pipelinea. Graphics principles are common for them.

[indigo zenith]

@sharp gazelleI'm trying to create my own simple implementation HLSL file for GPU Instancing shader but if I understood well, I need to make the shader incompatible with SRP batcher. The thing is, following the official document by adding a new material property without declaring it on the CBUFFER don't not only deactivate SRP compatibility but also throw an error saying that it is not compatible with SRP. Perhaps do you know why this error is raised ?

[sharp gazelle]

Why you need to make it SRP incompatible? Draw dispatching in GPU instancing is sole your task. You can do whatever you want in your shader. Can you describe your draw pipeline (preparation, draw dispatch, etc) in a more detail?

[indigo zenith]

I'm in a Entities graphics environment in URP 14.08

[indigo zenith]

Well first of all thx for your disponibility.
I'm at the begining of learning game dev world so I'm afraid I don't know the concept and semantic enough.
Well, I try to draw gizmo (lines and shapes built from lines).
I want to use Graphics.RenderMeshIndirect to draw those lines by passing a single cube mesh and a ComputeBuffer filled with matrices.
Now I need to create a custom shader to use GPU Instancing. I need to get access to unity_instanceID to draw the corresponding matrices.
I'm in a Entities graphics environment in URP 14.08

[sharp gazelle]

you don't need unity semantics. For getting instancei id you can use shader semantic:

VertexToPixelData VertexShader(uint instanceID: SV_InstanceID, (...other vertex input data...))

For Gizmos you shouldn't care about underlying SRP (URP/HDRP or even builtin), because you dont need any shading, fog application, etc for it

For example full vertex shader setup:

struct VertexInput
{
    float3 pos: POSITION;
    uint instanceID: SV_InstanceID;
};

struct VertexToPixel
{
    float4 pos: SV_Position;
    float4 color: COLOR0;
};

VertexToPixel VS(VertexInput i)
{
...
}

As you can see we can add instance id into input structure instead of separate field

[indigo zenith]

well i'm gonna try that, how do you send code formating message in discord ?

[sharp gazelle]

https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline-#h_01GY0DAKGXDEHE263BCAYEGFJA

Three backticks and "cs" for C#

[indigo zenith]

here is my testing shader

            HLSLPROGRAM

            #pragma vertex vert
            #pragma fragment frag
            #pragma multi_compile_instancing

            #include "Packages/com.unity.render-pipelines.universal/ShaderLibrary/Core.hlsl"
            #include "Packages/com.unity.render-pipelines.core/ShaderLibrary/UnityInstancing.hlsl"

            struct Attributes {
                float4 positionOS   : POSITION;
                #if UNITY_ANY_INSTANCING_ENABLED
                uint instanceID: SV_InstanceID;
                #endif
            };

            struct Varyings{
                float4 positionHCS  : SV_POSITION;
                #if UNITY_ANY_INSTANCING_ENABLED
                uint instanceID: SV_InstanceID;
                #endif
            };

            struct MeshProperties{
                float4x4 m;
            };

            StructuredBuffer<MeshProperties> _Properties;

            Varyings vert(Attributes IN){
                Varyings OUT;

                float4 pos = mul(_Properties[IN.instanceID].mat, IN.positionOS);
                OUT.positionHCS = TransformObjectToHClip(pos.xyz);

                return OUT;
            }

            half4 frag(Varyings IN) : SV_Target{
                return half4(1, 1, 1, 1);
            }

            ENDHLSL

but i get this error : A BatchDrawCommand is using the pass "<Unnamed Pass 0>" from the shader "Custom/InstancedIndirect" which does not define a DOTS_INSTANCING_ON variant.

[sharp gazelle]

How do you dispatch your draw?

[indigo zenith]

ahah hum, at the risk of looking like an idiot, I don't understand what do you mean by dispatch my draws, but i'm searching online to answer don't worry 😅

here is my system overview

protected override void OnUpdate()
{
    renderParams = new RenderParams(material);
    ...
    
    commandBuffer = new GraphicsBuffer
        (GraphicsBuffer.Target.IndirectArguments, commandCount, GraphicsBuffer.IndirectDrawIndexedArgs.size);
        
    UpdateBuffers();
    
    Graphics.RenderMeshIndirect(renderParams, mesh, commandBuffer, commandCount);
}

private void UpdateBuffers() // futur burst jobs
{
    // args buffer
    commandData = new GraphicsBuffer.IndirectDrawIndexedArgs[commandCount];
    commandData[0].indexCountPerInstance = (uint)mesh.GetIndexCount(0);
    commandData[0].baseVertexIndex = (uint)mesh.GetBaseVertex(0);
    commandData[0].instanceCount = (uint)count;
    commandData[0].startIndex = (uint)mesh.GetIndexStart(0);
    commandBuffer.SetData(commandData);

    // Instances
    var instances = new NativeArray<InstanceData>(count, Allocator.Temp);
    for (int i = 0; i < count; i++)
    {
        // set data
        instances[i] = data;
    }

    instancesBuffer = new ComputeBuffer(count, InstanceData.Size());
    instancesBuffer.SetData(instances);
    instances.Dispose();

    renderParams.matProps.SetBuffer("_Properties", instancesBuffer);
}

oh the error, is it possible that it is because I'm not using CGPROGRAM but HLSLPROGRAM ?

nah forget about it

[indigo zenith]

well don't mind actually, I will dig a lil bit deeper instead of wasting your time

[sharp gazelle]

Ah, sorry, you didn't reply to message so I forgot to look this thread. I am suspecting that new Graphics.Render* API functions work through BatchRenderGroup, and this is the reason why your shader don't working. Try to replace RenderMeshIndirect call with DrawMeshInstancedIndirect @indigo zenith

[indigo zenith]

Nah don't worry, I finally found my issue, by searching what you meant by "How do you dispatch your draw?" and actually the problem was that I assign this material to an entity that was render by the entities graphics pipeline, so using BatchRenderGroup. Now all is working fine with my simple HLSL file. You helped me a lot here. Thx again

[sharp gazelle]

Glad you have achieved your goal

[indigo zenith]

ah and just to mention I had to declare instanceID in the vert function because passing it in Attributes struct threw error "invalid subscript 'instanceID'" dunno why

            Varyings vert(Attributes IN, uint instanceID: SV_InstanceID)
            {
                Varyings OUT;

                float4 pos = mul(unity_InstanceData[instanceID].m, IN.positionOS);
                OUT.positionHCS = TransformObjectToHClip(pos.xyz);

                return OUT;
            }

[sharp gazelle]

#if UNITY_ANY_INSTANCING_ENABLED
uint instanceID: SV_InstanceID;
#endif

If it is wrapped like this, your non instanced shader variants will fail to compile

[indigo zenith]

yes I just changed that ^^'

[sharp gazelle]

you should wrap all usages of instanceID in shader body to prevent that

[indigo zenith]

so much to learn x)