#occuros's crash

1 messages · Page 1 of 1 (latest)

naive plank
royal venture
#

For reference I attached the log again

naive plank
#

yeah i mean i'm running out of ideas short of uploading the whole project

royal venture
#

Ok will start the process, its not humongous, but still a full game.

naive plank
#

thanks. i've asked the android people for future reference if there's some way i can get more tea leaves without you having to upload that, but i think they may have logged out for the day

#

the only other thing i can think of is if you can get a full crash dump out of the quest, that might actually be better in some ways than the project (because then we don't have to somehow have two quests going), but i am not currently up on how to extract crash dumps from android. i've asked around about that too

royal venture
#

There is also no thombstone created or similar

#

(which helped in the past when we identified the audio mixer issue on the quest)

naive plank
#

my new theory, as yet unsubstantiated, is that the netcode switch is actually not as irrelevant as it initially seemed; probably, some jobs get scheduled with deferred arrays differently and/or not at all depending on the default ghost mode

naive plank
#

netcode man asks: are you using self-hosting, i.e. client and server worlds both on the client that crashes? or the app is client-only? and, if you disable PhysicsSystemGroup, does it still crash?

#

(the idea is that probably the deferred job in question is a physics job)

#

@royal venture ^

royal venture
# naive plank <@163405522257575936> ^

It’s client server where the host is client + server and other join the host.

I will try to disable the physicssysremgroup as soon as the project is uploaded

It still is in the packing stage at the moment

#

We are also using relay to have the clients connect to the host

#

The client that crashes is the non hosting one.

#

Usually the host has no issue, it never crashed for the host

naive plank
#

ah that's interesting

#

@CristianMarastoni76 over here

#

@dense palm ^

dense palm
#

Hey!

#

so we have have only a couple of deferred array stuff in Netcode and it is mostluy for the server.

#

The only thing that defiitively trigger for the client (and only the client because you it is the one crashing) is that as soon as it has predicted ghosts:

  • physics run (if you have physics)
  • commands may be sent (deps on the version you have of Netcode. What it is ?)
#

This is why I asked to disable physics to try bi-secting a bit.

naive plank
#

hmm physics has this extension... ``` internal static class IJobParallelForDeferExtensionsPhysics
{
unsafe public static JobHandle ScheduleUnsafeIndex0<T>(this T jobData, NativeArray<int> forEachCount, int innerloopBatchCount, JobHandle dependsOn = new JobHandle())
where T : struct, IJobParallelForDefer
{
return IJobParallelForDeferExtensions.Schedule(jobData, (int*)NativeArrayUnsafeUtility.GetUnsafeBufferPointerWithoutChecks(forEachCount), innerloopBatchCount, dependsOn);
}

    //@TODO: Should this be NativeReference.ReadOnly?
    unsafe public static JobHandle ScheduleUnsafe<T>(this T jobData, NativeReference<int> forEachCount, int innerloopBatchCount, JobHandle dependsOn = new JobHandle())
        where T : struct, IJobParallelForDefer
    {
        return IJobParallelForDeferExtensions.Schedule(jobData, (int*)NativeReferenceUnsafeUtility.GetUnsafePtrWithoutChecks(forEachCount), innerloopBatchCount, dependsOn);
    }
}```
royal venture
dense palm
#

ok, so really recent. At least I'm looking at the same code base.

#

the NativeArray<int> forEachCount look suspicious but not necessaritly bad. Depends on the state of the forEachCount count indeed. But does this get patched ?
I would expect crashing way before the job is scheduled.

royal venture
#

When the crash happens not all ghosts have been sent to the client, it stops around half way.

#

(Not sure if that is any help)

naive plank
#

the schedule method does not get patched. and the crash in this case is actually at execute time, not at schedule time

dense palm
#

(because in the Editor works fine if you connect to the client/server build as a client)

royal venture
#

That said with the most recent beta release of the editor it became a lot more frequent for the crash.

#

Also one thing to mention is the entity which somehow influences the crash is not physics based

#

(One of the few things which aren’t physics based)

dense palm
#

When the crash happens not all ghosts have been sent to the client
Tell me nothing particular thinking about it. The only thing I can think of is that when it crash the server sent enough ghosts of certain kind (i.e predicted).
You mentioned that without predicted ghosts no crashs occurs.
Can you just force to have ghosts that are predicted to the client to be sent (via relevancy for example)

#

Also one thing to mention is the entity which somehow influences the crash is not physics based
That fine, as long as it is a predicted ghost, the prediction loop start running. Until you received a prediction ghost, the PredictedSimulationSystemGroup does not run, nor will do the PredictedFixedStepSimulationSystemGroup (and so physics).
Did you setup the physics group to run all the time or are you still running it with the usual default (only if on the client exist at least one entity with physics velocity and/or lag compensation is enabled) ?

royal venture
#

(Lag compensation enabled or Kinmatic ghosts) from the netcode physics settings

#

Should it try “always run”?

dense palm
#

You can force it. That may trigger the issue time 0.
If you do so, you need to also force the PredictedSimulationSystemGroup to run all the time, via ClientServerTickRate.PredictionLoopUpdateMode.
You can enable that via NetcodeConfig if you have one.

royal venture
#

Making a build now to test with physics always on

royal venture
#

(to clarify we spawn a client/server worlds on startup for the first scene)

#

Not sure what to do now...

#

Ok now I also get an error in the editor

dense palm
#

oh!!! That great!!

#

it may help finding the thing faster.

royal venture
# 
        internal static unsafe JobHandle ScheduleUnityPhysicsBodyPairsJob<T>(T jobData, Simulation simulation, ref PhysicsWorld world, JobHandle inputDeps)
            where T : struct, IBodyPairsJobBase
        {
            SafetyChecks.CheckSimulationStageAndThrow(simulation.m_SimulationScheduleStage, SimulationScheduleStage.PostCreateBodyPairs);

            var data = new BodyPairsJobData<T>
            {
                UserJobData = jobData,
                PhasedDispatchPairs = simulation.StepContext.PhasedDispatchPairs.AsDeferredJobArray(),
                Bodies = world.Bodies
            };

            var jobReflectionData = BodyPairsJobProcess<T>.jobReflectionData.Data;
            BodyPairsJobProcess<T>.CheckReflectionDataCorrect(jobReflectionData);

            var parameters = new JobsUtility.JobScheduleParameters(UnsafeUtility.AddressOf(ref data), jobReflectionData, inputDeps, ScheduleMode.Single);
            return JobsUtility.Schedule(ref parameters);
        }
#

thats the physics job which creates the issue

#

This is our system that triggers the job

#

I can't spot what we are doing wrong at first glance, but disabling that system removes the crash in editor

sterile sleet
#

i might know issue give me 1 sec, i need to confirm in a non-forked version of physics

#

can you just try something quickly, in your system add a dependency to BroadphaseData

#

ah it's internal

#

what was the actual error sorry?

royal venture
#

Its a nullpointer dereference

#

coming from the system: ScheduleUnityPhysicsBodyPairsJob

sterile sleet
#

you have an editor log now though right? what line specifically

royal venture
#

Yep will tell you in a minute, just making a builld with that system disabled to double check it also prevents the crash on the device

# 
NullReferenceException: Object reference not set to an instance of an object
Unity.Collections.NativeList`1[T].AsDeferredJobArray () (at ./Library/PackageCache/com.unity.collections@56bff8827a7e/Unity.Collections/NativeList.cs:702)
Unity.Physics.IBodyPairsJobExtensions.ScheduleUnityPhysicsBodyPairsJob[T] (T jobData, Unity.Physics.Simulation simulation, Unity.Physics.PhysicsWorld& world, Unity.Jobs.JobHandle inputDeps) (at ./Library/PackageCache/com.unity.physics@b42dcf5fc3ad/Unity.Physics/Dynamics/Simulation/IBodyPairsJob.cs:132)
Unity.Physics.IBodyPairsJobExtensions.Schedule[T] (T jobData, Unity.Physics.SimulationSingleton simulationSingleton, Unity.Physics.PhysicsWorld& world, Unity.Jobs.JobHandle inputDeps) (at ./Library/PackageCache/com.unity.physics@b42dcf5fc3ad/Unity.Physics/Dynamics/Simulation/IBodyPairsJob.cs:93)
Holonautic.Scripts.Experiments.DisableNonDynamicPairsSystem.OnUpdate (Unity.Entities.SystemState& state) (at Assets/Holonautic/Scripts/Systems/MixedSystems/PhysicsSystems/DisableNonDynamicPairsSystem.cs:33)
royal venture
sterile sleet
#

yeah thats fine

#

i know what list it is

sterile sleet
#

unfortunately my theory is wrong

#

oh wait no its totally correct

#

i just had errors hidden🙃

#

if there are 0 dynamic physics objects in the world

#

then PhasedDispatchPairs is null

image.png
#

and throws an exception

#

if you want to fix this, fork physics, IBodyPairsJob

            where T : struct, IBodyPairsJobBase
        {
            if (!simulation.StepContext.PhasedDispatchPairs.IsCreated)
            {
                return inputDeps;
            }
            SafetyChecks.CheckSimulationStageAndThrow(simulation.m_SimulationScheduleStage, SimulationScheduleStage.PostCreateBodyPairs);

            var data = new BodyPairsJobData<T>
            {
                UserJobData = jobData,
                PhasedDispatchPairs = simulation.StepContext.PhasedDispatchPairs.AsDeferredJobArray(),
                Bodies = world.Bodies,
            };

Add this early out

#

alternatively, if you dont want to fork physics you could probably avoid ti in your system by doing something like
state.RequireForUpdate<PhysicsVelocity>()

#

or maybe
if(physicsWorldRef.ValueRO.CollisionWorld.NumDynamicBodies == 0) return;

#

i haven't tested these, i only tested fixing ScheduleUnityPhysicsBodyPairsJob

royal venture
dense palm
#

Oh my! That looks cool! Glad we find the cultprit of that. And great! It wasn't NetCode 😛

royal venture
dense palm
#

I think it is something relatively recent, because it was not occurring before. We are poking the physics dudes!

sterile sleet
royal venture
sterile sleet
#

saves you forking

royal venture
#

Thank you everyone for helping tracking that down, and having a very good workaround until it can be addressed in the package!

sterile sleet
#

hopefully that's the end of your troubles and gl on your pending launch ^^

royal venture
onyx spindle
#

Good job figuring this out so quickly. Sorry I am late to the party. Was busy with another emergency that came in before that 😅

royal venture
#

@dense palm we currently have an issue of:

InvalidOperationException: Could not find ghost type in the collection
Unity.NetCode.PredictedGhostSpawnSystem+InitGhostJob.Execute (Unity.Entities.ArchetypeChunk& chunk, System.Int32 unfilteredChunkIndex, System.Boolean useEnabledMask, Unity.Burst.Intrinsics.v128& chunkEnabledMask) (at Library/PackageCache/com.unity.netcode@e1dc18e490f0/Runtime/Snapshot/PredictedGhostSpawnSystem.cs:203)

The ghost type in question is always the same number, but how can we know what this could relate to?

It is also not deterministic, sometimes the error is thrown and sometimes not.

Is there anyway we can get more information to pinpoint what the problem could be?

dense palm
#

@royal venture hey! sorry Missed that.
The reason this is occurring is because you are predicted spawning a ghost prefab on the client. And that prefab does hasn't been processed yet by the GhostCollectionSystem.
That because the client didn't received from yet from the Server the prefab list, that the GhostCollection populate and map to the client loaded prefabs.

Because the client don't assign the prefab to the GhostCollectionPrefab buffer until the he receive the entries, that buffer may not contain a ghost for the type (GhostType component) assigned to the prefab.

And this is the error the Unity.NetCode.PredictedGhostSpawnSystem is triggering.

The radominicy it because all this depends on:

  • Network connectivy
  • Scene loading timing
  • number of prefabs
royal venture
dense palm
#

Yes there is: use the GhostCollection.GhostTypeToColletionIndex. This map contains a GhostType to index (int) entry if the prefab has been assigned.

royal venture
#

@dense palm not sure if that would be sindered an issue, but if the user adds an entity to a LEG (Linked Entity Group) but then does destroy the entity without removing it from the LEG the following error is thrown:

#

Which, was not very evident where the issue lies (even with journaling enabled).

📎 message.txt