SourceGen failed to generate IJE schedule code from my generated System

[limber leaf]

As said in titile, I got an exception that point to Job scheduling code (see in picture)

new TweenJob // <- Here
{{
    DeltaTime = SystemAPI.Time.DeltaTime,
}}.Run();

Thanks for reading.

[hushed cobalt]

can you show your components

side not you shouldn't include this attribute

            [Unity.Burst.BurstCompile]
            void Execute(```

only need it on the struct

[limber leaf]

namespace Components.Camera
{
    public struct AddPosTweener_TweenData : Unity.Entities.IComponentData
    {
        public float LifeTimeSecond;
        public float BaseSpeed;
        public Unity.Mathematics.float3 Target;
    }
}

namespace Components.Camera
{
    public struct Can_AddPosTweener_TweenTag : Unity.Entities.IComponentData, Unity.Entities.IEnableableComponent
    {
    }
}

using Unity.Entities;
using Unity.Mathematics;

namespace Components.Camera
{
    public struct AddPos : IComponentData
    {
        public float3 Value;
    }
}

here

[hushed cobalt]

ok so those are about as basic a component as you can get

which really only leaves AddPosTweener if anything weird is going on there?

no statics or anything?

[limber leaf]

Really? I thought we had to put BurstCompile on it was just as we do with methods in ISystem

[hushed cobalt]

nope

only required on top of jobs

[limber leaf]

using Unity.Entities;

namespace SomeNamespace
{
    public interface ITweener<Component, Target>
        where Component : unmanaged, IComponentData
        where Target : unmanaged
    {
        void Tween(ref Component componentData, in float baseSpeed, in Target target);

        bool CanStop(in Component componentData, in float lifeTimeSecond, in float baseSpeed, in Target target);
    }

}

[BurstCompile]
public partial struct AddPosTweener : ITweener<AddPos, float3>
{
    [BurstCompile]
    public bool CanStop(in AddPos componentData, in float lifeTimeSecond, in float baseSpeed, in float3 target)
    {
        const float epsilon = 0.01f;
        return math.all(math.abs(target - componentData.Value) < new float3(epsilon));
    }

    [BurstCompile]
    public void Tween(ref AddPos componentData, in float baseSpeed, in float3 target)
    {
        componentData.Value =
            math.lerp(componentData.Value, target, baseSpeed * this.DeltaTime);
    }

}

[limber leaf]

nope, AddPosTweener is normal helper struct

[hushed cobalt]

hmm looks fine

[hushed cobalt]

oh to complicate it even more, you don't need it on the top of ISystem anymore ^^

(not that this is breaking anything, just code cleanliness)

[limber leaf]

... First time know this

[hushed cobalt]

it was required until like 2 years ago

but they made it so isystem didn't require it a while ago

[limber leaf]

and forgot to mention, first 2 components are auto generated.

[hushed cobalt]

oh are they source generated?

because if so you can't use generated components in other source generation (ijobentity)

that basically explains your problem

[limber leaf]

oh... so my only solution is learning IJobChunk?

[hushed cobalt]

if you want to use source generated components yes you'd need to use ijobchunk

[languid epoch]

Have to put them on another assembly if you want to use IJE. On the same assembly, source generators do not know each other

[limber leaf]

As far as I know, generated code is compiled into same assembly as the original source code, the only reference I could find is generate-source-based-on-other-assembly and seems it has nothing to do with put the generated code into another assembly

I am currious why ISystem was generated in my above code but IJE wasn't

[languid epoch]

This doesn't contradict to what I said above. 😄

If you have an "AssemblyA" and 2 generators working on it, then each generator doesn't know the output code of the other.

To simplify things, you can think each generator has to do call this method Type.GetAllTypesInThisAssembly() to get all type info avaible within AssemblyA

But the type array returned from that method is the same for the 2 generators.

It never includes additional types generated by those 2 generators.

To solve this problem, the only way is to generate some additional types into another assembly that is the dependency to your AssemblyA

That assembly will be compiled before AssemblyA, so generators working on AssemblyA can pick up the additional types.

In your case, you have to put generated components into another assembly so your system assembly can pick them up

For the past projects, I usually have 1 assembly for the components and another for the systems.

[limber leaf]

So two generators for both components and systems assembly, I guess?

[languid epoch]

I think you misunderstand me

To sum up, if you want IJobEntity generator knows about your generated components, you have to generate them into another assembly. Not in the same assembly with your IJE

[languid epoch]

I'd just assume you understood the issue incorrectly

[limber leaf]

The first 2 components, the ISystem and the Job are all generated by my only one generator, so they must are in the same assembly? I can see that the ISystem was generated correctly (I saw it in Systems tab in the editor).
I will call it AssemblyA that contains all my generated code above.
At first (before above code generated), IJEGenerator & ISystemGenerator & MyGenerator will work on AssemblyA -> IJEGenerator & ISystemGenerator should not know about generated code from MyGenerator -> both of them won't generate any code -> ISystem should not have been generated

[languid epoch]

does the generated code of your system really uses those 2 components?

You have to inspect the generated code, not the System window

also, what part of the component is generated, what is not?

for example:

public partial struct MyComponent : IComponentData, IMakeThisAnAttribute { }

if the generated code of system only uses the component type name, then it's fine. Because the type name is defined manually in the assembly already.

But if it is going to use the generated parts of this MyComponent then the ISystem generator will fail

This is fact because it's the way roslyn source generator is designed by the .NET team

there is no way to circumvent this limitation

[limber leaf]

Ok, after one day of thinking, confusion is still there. Now I will explain what was I about to do and hope you can tell me if it is doable, I think I misunderstanding the purposes of source generator then make wrong code structure.

My current project structure:

• TweenLib

• Components

• Utilities (depends on Components & TweenLib)

• Authoring (depends on Utilities & Components)

• Systems (depends on Utilities & Components)

• ITweener<ICD, Target> live in my TweenLib assembly

• Concreted Tweener live in Utilities assembly (users define their own tweener)

• I have a source gen: Find every Tweener and generate 2 new ICDs + 1 System for each of it (generation based on Tweener name, ITweener arguments' names and namespaces)

Source gen based on Syntax node of Tweener -> Of course, generated codes will lay in Utilities assembly (which make... components and systems lay in Utilities?? feel so wrong) -> problem occurred as I posted.

[languid epoch]

Instead of explaning by words, why don't you make a reproducible github repo? It would be much more clearer to people what's your intention.

It's fine if you're not willing to share your actual code, especially your source generator code.

But at least you should represent your Unity project structure in the simplest form (with .asmdefs for your assemblies).

Then some pseudo code here and there to demonstrate the setup of your manually written code, and the generated code resulted from that setup.

It matters to know where things locate.

[limber leaf]

I have no problem with sharing my code for now, I just thought that my previous message was clear enough. I will make a github repo rightaway

[languid epoch]

Explaining your issue by words requires too much of people's mental power and assumptions which could be very wrong.

Only use words as the last resort when you don't have any reproducible code at hand.

[limber leaf]

Here you go Github
I made 2 commits, first one for workable version and second one for non-workable, details in commit description.

[languid epoch]

Apparently this generated code will never work

[Unity.Burst.BurstCompile]
public partial struct TweenJob : IJobEntity
                                 ^^^^^^^^^^
{
    [Unity.Collections.ReadOnly] public float DeltaTime;

    [Unity.Burst.BurstCompile]
    void Execute(
          ...
        )
    {
        ....

    }

}

Also this

[Unity.Burst.BurstCompile]
public void OnUpdate(ref SystemState state)
{
    new TweenJob
    {
        DeltaTime = SystemAPI.Time.DeltaTime,
                    ^^^^^^^^^
    }.ScheduleParallel();
        
}

Because IJobEntity generator and SystemAPI generator can never know about them.

Both those generators and your own ClassLibraryGenerator operates on the same Utilities.asmdef assembly.

Let me state this again: On the same assembly, source generators do not know each other. They receive the same compilation context (ie your manually written code as well as APIs from referenced assemblies) then operate on that context independently.

Generators in group A receives the same things in group P and R.

But generators in group A can never know, can never be able to read into these generated code

JobEntityGenerator and SystemGenerator can never expand the code in TransformPositionTweener_TweenSystem.g.cs file

Because at the beginning of compilation (where generators run), this file does not exist in the compilation context.

On the other hand, take the Systems.asmdef as an example. Because this SetCubeMoveLeftSystem.cs file already exists before the compilation starts. SystemGenerator can expand the SystemAPI.Query on the left into the code on the right

[languid epoch]

Let's say, your Utilities.asmdef originally has 3 code files:

• 2 code files that use SystemAPI will be processed by SystemGenerator.
• 2 code files that use IJobEntity will be processed by JobEntityGenerator.
• 1 code file declares a TransformPositionTweener struct so it will be processed by "TweenGenerator". Your TweenGenerator will emit an additional code file according to the code of TransformPositionTweener struct.

So you see, because generators work independently to each other, SystemGenerator and JobEntityGenerator never know about that additional code file written by your TweenGenerator.

If that additional file includes code must be processed by them, they can never work on it.

When an .asmdef is finally compiled into a .dll it's already too late.

[languid epoch]

Did you think that by making Utilities.asmdef a reference of Systems.asmdef, SystemGenerator and JobEntityGenerator will have a chance to work on your generated code inside Utilities.asmdef?

It never works that way though.

What is actually referenced by Systems.asmdef is the already compiled Utilities.dll, not the source code inside Utilities.asmdef

By design, Roslyn source generators cannot modify the content of a compiled .dll.

They can only work with source code, manually written by us, to emit more code before the whole assembly is going through the C# compiler.

[limber leaf]

This is the first time I've known we could see generated code like this in VS

[limber leaf]

Ok, I think I understand all you just said, I still have some questions though:

1. Can you tell me if I understand the following picture correctly? In my perspective, the steps in the pic are: All original code in certain assembly got compiled -> source generators get syntax tree of the assembly -> source generators work -> Add generated code to the compiler -> compilation done with both original and generated code compiled.
2. Any hint to help me out of my situation? I am still not able to wrap my head around the idea the only way is to generate some additional types into another assembly that is the dependency to your AssemblyA for my case.

[limber leaf]

3. We can't modify the content of a compiled .dll but we can extend it, right? Like a partial class in compiled dll can be extended by another partial class in assembly that reference the compiled dll

[languid epoch]

1. Correct
2. I said that before knowing that you generated the whole system that uses SystemAPI and the whole IJE. Before the repo was provided, I thought that you wrote the system and IJE manually, but their code uses some functionality generated into the TransformPositionTweener struct.

So for that case I thought about, on the same assembly, this never works:

// Supposedly these code are manually written
// inside an assembly called TweenSystems.asmdef

public partial struct TweenJob : IJobEntity
{
    void Execute(ref TransformPositionTweener tweener)
    {
        tweener.SomeGeneratedMethod();
    }
}

public partial struct TransformPositionTweener : ITweener { }

For this case to work, you have to put TransformPositionTweener in another assembly so it can be compiled before TweenSystems.asmdef.

But your actual case can never be achieved by any mean, I'm afraid you're at a dead end.

What you want to achieve can only be possible if Roslyn has a mechanism to run generators in an order. But the current design doesn't provide that capacity. Even the .NET team seems to be not interested in making it possible.

Their argument is that such mechanism will complicate things much more than we naively perceive and can make the IDE unusable.

[languid epoch]

3. What do you mean by "partial class be extended by another partial class"? It sounds very wrong to me. Remember that "partial" only has effect for source code so that you can split the source over multiple physical files. But they have to stay inside the same assembly. But after compilation, "partial" just doesn't exist anymore in the .dll.

// AssemblyA.asmdef
public partial MyClass
{
    public void Method1() { }
}

// AssemblyB.asmdef
public partial MyClass
{
    public void Method2() { }
}

This becomes 2 separated classes which have the same name MyClass and you will receive an "ambiguity error".

[limber leaf]

ok got it, thought we could spread partial over assemblies.

[languid epoch]

Any hint to help me out of my situation?

For job, you have to generate IJobChunk. For system you have to generate completely everything.

You can inspect the code gened by SystemGenerator to get a sense what you'll need for the final form of a system.

[limber leaf]

what about the method described in this link? It about generating code from compiled assembly, I haven't tried it so dunno if it could work with generated code from compiled assembly

[languid epoch]

Which method?

[languid epoch]

No, it's just not what you thought. You misunderstood its purpose.

Again, source generators cannot modify compiled code. Period. End of story.

The only technique that allows you to modify compiled dlls is IL Weaving. But it's a very advanced technique for you have to dig into IL code.

[limber leaf]

well, . _. better not touch IL Weaving for now