Source Generators and asmdef Scoping problems! | Official Unity Discord | Page 1

outer mirage Dec 1, 2022, 9:21 PM

#

Heya, anyone have any experience with source generators under an amsdef?

I can't seem to get source generators scoped and working under an assembly as described in the unity docs, was hoping someone had some insight

sullen rose Dec 1, 2022, 9:27 PM

#

what are you trying to use source generators for? they are not well supported in unity

outer mirage Dec 1, 2022, 9:29 PM

#

Trying to generate a static class that holds associative information/static methods that can be retrieved by a string ID

sullen rose Dec 1, 2022, 9:29 PM

#

this is an xy problem. what are you trying to do

outer mirage Dec 1, 2022, 9:29 PM

#

Essentially making a few big switch statement that are compatible with burst/jobs

sullen rose Dec 1, 2022, 9:30 PM

#

are you open to a different approach

#

to whatever it is you are trying to do

#

99% likely you do not need source geneators / you should not be using source generators

outer mirage Dec 1, 2022, 9:30 PM

#

Yes, as long as its compile time static and / or job compatible

sullen rose Dec 1, 2022, 9:30 PM

#

okay so what is the issue

#

what are you trying to solve

outer mirage Dec 1, 2022, 9:33 PM

#

I essentially need a generic interface hashmap that works with the burst/jobs system

#

But since native containers don't support managed interfaces, and using generics means writing out a new handler for every one of potentially 30-40 discrete bundles of otherwise similar types

#

I was hoping to get away with generating one or more switch statements that linked fired all the like static methods based on the given string id

sullen rose Dec 1, 2022, 9:36 PM

#

outer mirage I essentially need a generic interface hashmap that works with the burst/jobs sy...

what do you mean?

#

for what?

#

what are you trying to do

#

what is the actual thing you are doing

outer mirage Dec 1, 2022, 9:36 PM

#

Running arbitrary functions from a job

#

Well, "arbitrary" being one of a predefined at compile-time set

sullen rose Dec 1, 2022, 9:37 PM

#

okay but why

#

but since native containers don't support managed interfaces, and using generics means writing out a new handler for every one of potentially 30-40 discrete bundles of otherwise similar types
what are you trying to do

#

for example, "this is a speech synthesis library"

#

i haven't heard an issue yet either

#

lol i'm sorry i'm not trying to be annoying

outer mirage Dec 1, 2022, 9:39 PM

#

I'm creating a system that handles reading out data from XML into DOTS during runtime based on an arbitrary number of files, and would like to be able to parrallellize the logics for loading in each components data instead of having to do in linearly in managed code vs burst or IL2CPP

sullen rose Dec 1, 2022, 9:41 PM

#

hmm okay

outer mirage Dec 1, 2022, 9:41 PM

#

Since each component has a unique common ID ("com.example.first_example"), it can be tied to a discrete bundle of creation logic for different contexts (Managed Code, Linear Job, Parallel job)

sullen rose Dec 1, 2022, 9:41 PM

#

it sounds like you are trying to program using xml

#

does this sound right?

outer mirage Dec 1, 2022, 9:42 PM

#

when I say component here, I don't mean DOTS component, I mean my own designs "component"

#

No

sullen rose Dec 1, 2022, 9:42 PM

#

hmm

#

what is the source for your source generator?

outer mirage Dec 1, 2022, 9:42 PM

#

Me

sullen rose Dec 1, 2022, 9:42 PM

#

what kind of file is it

#

is it an xml file

outer mirage Dec 1, 2022, 9:42 PM

#

Wait, what?

#

XML has nothing to do with the built step

sullen rose Dec 1, 2022, 9:43 PM

#

hmm

#

okay can you show me the first 3 lines of your Execute method of your ISourceGenerator

#

just like a little snippet

#

and also, show me all the imports (the using) at the top?

#

this is to help me understand what you are trying to do

#

because what you described is really abstract

outer mirage Dec 1, 2022, 9:45 PM

#

using System.Text;
using System.Collections.Generic;
using System.Diagnostics;

using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.Text;
using Microsoft.CodeAnalysis.CSharp.Syntax;

        public void Execute(GeneratorExecutionContext context)
        {
            if (context.SyntaxReceiver is SyntaxReceiver syntaxReceiver) { }
            else return;

            StringBuilder typemapBody = new StringBuilder(),
                          gettypesBody = new StringBuilder();

            foreach (var keyValue in syntaxReceiver.ComponentStructsToAdd)
            {
                typemapBody.Append(StructToSwitchCase(keyValue.Key, keyValue.Value));
                gettypesBody.Append(IDStringToArrayItem(keyValue.Key));
            }

            context.AddSource(
                OutputFileIdentifier,
                SourceText.From(
                    WrapSourceBody(
                        WrapTypemapMethodBody(
                            typemapBody.ToString()
                        ) +
                        WrapGettypesMethodBody(
                            gettypesBody.ToString()
                        )
                    ),
                    Encoding.UTF8
                )
            );
        }

sullen rose Dec 1, 2022, 9:45 PM

#

okay

outer mirage Dec 1, 2022, 9:45 PM

#

This is even just the test, I was more getting balked at the asmdef scoping issue then it not working

#

The docs specifically call out a source generator should only run on the defined library, or libraries that reference it if its under an asmdef, but ime it never runs at all under one, regardless of references or not

sullen rose Dec 1, 2022, 9:46 PM

#

ComponentStructsToAdd, what is this?

#

does this code compile?

outer mirage Dec 1, 2022, 9:48 PM

#

I chopped up my code to the parts you asked, and yes it compiles and runs, and generates working code, just not under the asmdef like the docs say its supposed to

sullen rose Dec 1, 2022, 9:48 PM

#

just checking. it's a simple yes or no

#

okay

#

so ComponentStructsToAdd, you authored right?

outer mirage Dec 1, 2022, 9:50 PM

#

Sorry, I'm not trying to be unapprecative, I do appreciate the help, I've just been stuck on this scoping thing for days now, and its such a frusterating issue because I have no idea if its a bug, bad support for this in general, if I'm doing something wrong, or what, and its blocking moving forward on the rest of my project

#

Yes

sullen rose Dec 1, 2022, 9:51 PM

#

okay

outer mirage Dec 1, 2022, 9:51 PM

#

its just a dictionary of <string, StructDeclerationSyntax>

sullen rose Dec 1, 2022, 9:51 PM

#

so it really depends what your goals are

#

it sounds like you "just" want this to work

#

so i think you can find some examples of source generators in unity and sort it out

outer mirage Dec 1, 2022, 9:52 PM

#

My goal is to have a set of methods that can be sorted and run by a string id in a burst context

sullen rose Dec 1, 2022, 9:52 PM

#

but the fact that you're doing that is bad

#

it means you are doing something very wrong

outer mirage Dec 1, 2022, 9:52 PM

#

Why though?

#

Why is it bad to load user defined data into runtime via compile time functions?

sullen rose Dec 1, 2022, 9:52 PM

#

that's a very subjective assessment, but i'm not wrong

#

hmm

#

it's bad to do dynamic dispatch with strings for what sounds like a kind of murky and abstract purpose

outer mirage Dec 1, 2022, 9:53 PM

#

There fixed at 64 bytes long

sullen rose Dec 1, 2022, 9:53 PM

#

i'm not sure why you would want to use Jobs at all

#

hmm

#

the performance characteristics are not really the concern

#

for something that is going to take ~~milliseconds~~ microseconds either way

outer mirage Dec 1, 2022, 9:54 PM

#

Because some of the datasets can have a few hundred components to load, and since you can do it in paralell, and they are all self contained, why not?

sullen rose Dec 1, 2022, 9:54 PM

#

datasets of what exactly

#

components and datasets are very abstract

#

it sounds like you are loading an xml file, then creating objects in a unity scene

outer mirage Dec 1, 2022, 9:55 PM

#

<Component type="com.example.example_one">
  <ComponentSettings>
    <ComponenetSetting name="height">1.0</ComponentSetting>
  <ComponenetSettings/>
<Component/>

#

It could be JSON

#

It could be a serialized binary file

#

XML is just the random serialization format I chose

sullen rose Dec 1, 2022, 9:55 PM

#

okay well

#

i guess my last piece of feedback here is i'm not sure if this stuff makes sense, but i think you'll be able to figure it out

outer mirage Dec 1, 2022, 9:56 PM

#

I just don't understand what the hostility to the method is?

rotund onyx Dec 1, 2022, 9:57 PM

#

outer mirage Heya, anyone have any experience with source generators under an amsdef? I can...

Did you build the source generator to dll and label with RoslynAnalyzer ?

#

What do you mean by "under an asmdef"

outer mirage Dec 1, 2022, 9:58 PM

#

Yes, and if I don't have it under an asmdef it runs just fine, but then its a global one that runs against every assembly,

#

Like, a dedicated assembly def in unity, a .asmdef file

sullen rose Dec 1, 2022, 9:59 PM

#

outer mirage I just don't understand what the hostility to the method is?

it's not hostility so much as it is a bad idea. or at least, it's hard for me to say with 100% intellectual honesty

#

until i know concretely what this is for

#

but like i said, i am probably right...

#

sorry

outer mirage Dec 1, 2022, 9:59 PM

#

I don't know how better to explain it, and that right there is the needless hostility

sullen rose Dec 1, 2022, 10:00 PM

#

i guess i have limited time here i just wanna cut straight to the chase. i could suggest a better approach maybe

#

but i have to understand what you are trying to do

#

what is all of this for? what is the game?

rotund onyx Dec 1, 2022, 10:00 PM

#

outer mirage Yes, and if I don't have it under an asmdef it runs just fine, but then its a gl...

If you put dll with asmdef what happens? Your sourcegen doesn't run?

outer mirage Dec 1, 2022, 10:01 PM

#

rotund onyx If you put dll with asmdef what happens? Your sourcegen doesn't run?

It does not seem to run, If I remove the asmdef file, it works like expected but then I'm out an asmdef that I created for a reason

outer mirage Dec 1, 2022, 10:02 PM

#

sullen rose what is all of this for? what is the game?

Its for an internal library of stuff I"m working on for several games I'm working on

sullen rose Dec 1, 2022, 10:02 PM

#

hmm

#

well would you be interested in a better approach? i asked you early on

#

it sounds like the answer is no

#

which is okay

#

unity isn't really designed to run source generators

outer mirage Dec 1, 2022, 10:02 PM

#

But you didn't explain why it was better

sullen rose Dec 1, 2022, 10:02 PM

#

but you can create a dotnet package, in visual studio

#

and you can use source generators there

outer mirage Dec 1, 2022, 10:03 PM

#

You just said mine was bad, and then didn't give an alternative to the problem

sullen rose Dec 1, 2022, 10:03 PM

#

outer mirage You just said mine was bad, and then didn't give an alternative to the problem

hmm, but i don't understand what this is for yet

#

for example, "i am trying to make a web interface for building levels for an RPG, and the site saves a map file that is later loaded by unity. the maps are large, so i am trying to parallelize the loading of this file format."

rotund onyx Dec 1, 2022, 10:05 PM

#

outer mirage It does not seem to run, If I remove the asmdef file, it works like expected but...

Hm. Maybe it's due to Unity bug, or your code failed to generate source
How is your folder structure and assembly references?
Also for last resort, you can consider make it global but filter with currently compiling assembly name instead 😌

#

Sourcegen support for Unity is pretty early stage rn

outer mirage Dec 1, 2022, 10:07 PM

#

sullen rose for example, "i am trying to make a web interface for building levels for an RPG...

I have an pre-defined pre-verified with a schema XML file -> I want to load data from this XML file into a normalized struct -> I want to scan the now created set of normalized structs and use the commonID each one has to run a specific static function that takes an entity reference, the proper entitymanager/commandbuffer for the context (managed, job, and paralell job), and the normalized data struct, and produces an output entity with unity ecs components filled with the xml data

sullen rose Dec 1, 2022, 10:10 PM

#

outer mirage I have an pre-defined pre-verified with a schema XML file -> I want to load dat...

can you give a concrete example for what this is meant for? for example, "i described a scene of a natural landscape in xml, and i want to load it as a bunch of ECS entities for rendering and simulating this natural landscape"

#

it doesn't sound like this loading process is performance sensitive so i don't know why it should be a job

#

you can write ordinary c# code to create the appropriate entities

#

but maybe focus on the concrete example for a sec

#

it's been almost 40 minutes

#

and i still don't really know what this is for

outer mirage Dec 1, 2022, 10:12 PM

#

That IS what its for, loading in arbitrary XML data into ECS! Like, Its arbitrary what it can used to make; If I set it up for bumpers, flippers, a table, and whatever and the proper unity ecs components/system, its for a pinball game; if I set it up for car parts, its for racing

sullen rose Dec 1, 2022, 10:13 PM

#

okay but what is the 1 first concrete game

#

hmm

outer mirage Dec 1, 2022, 10:16 PM

#

sullen rose okay but what is the 1 first concrete game

Are you asking for the marketing copy for the project? The idea is to allow for things to be published by either me, or other players in anything via a simple serialization format, the first game I'm working on is pinball table loading

sullen rose Dec 1, 2022, 10:16 PM

#

the answer to that question should be really succinct

#

like "it's a racing game"

outer mirage Dec 1, 2022, 10:16 PM

#

Its a pinball game

sullen rose Dec 1, 2022, 10:16 PM

#

okay great

outer mirage Dec 1, 2022, 10:16 PM

#

Then I'm using the same system for others

sullen rose Dec 1, 2022, 10:16 PM

#

so these are maps for a pinball game?

outer mirage Dec 1, 2022, 10:17 PM

#

They are table definitions, amongst other potentials

sullen rose Dec 1, 2022, 10:17 PM

#

and you want other players to create pinball maps

outer mirage Dec 1, 2022, 10:17 PM

#

For another game, they are much more varied and arbitrary

sullen rose Dec 1, 2022, 10:17 PM

#

can i give you a piece of feedback?

#

without you getting mad?

#

this is an xy problem. what are you trying to do
the answer to this question i asked was, "i am trying to make a pinball game where the players can make the pinball maps"

outer mirage Dec 1, 2022, 10:18 PM

#

Ok, the problem is is you then will suggest something simper and more direct/linear in approach

sullen rose Dec 1, 2022, 10:18 PM

#

am i?

outer mirage Dec 1, 2022, 10:18 PM

#

You already did

sullen rose Dec 1, 2022, 10:19 PM

#

you're syaing i'll suggest a better approach, which i will

#

like i said it's subjective. but what you're doing doesn't make sense really

outer mirage Dec 1, 2022, 10:19 PM

#

For one project, yes it does not make sense

sullen rose Dec 1, 2022, 10:19 PM

#

anyway. there is a package authored by unity that tries to tackle this problem. it's called com.unity.runtime-scene-serialization

outer mirage Dec 1, 2022, 10:19 PM

#

As the backbone to more then one project, with the intent of maturing it through several, it makes more sense

sullen rose Dec 1, 2022, 10:20 PM

#

i think if you are really careful and truly make a fully decoupled architecture for the maps, it is possible this package will Just Work for you

#

i think this is a hard problem. nobody is going to author a pinball map in a text file

outer mirage Dec 1, 2022, 10:20 PM

#

No they wont, its a serialization format

sullen rose Dec 1, 2022, 10:20 PM

#

so i think you should look at com.unity.runtime-scene-serialization

#

it is sort of like a working, more thoughtful version of what you are trying to do

outer mirage Dec 1, 2022, 10:21 PM

#

Thats for loading scenes

#

I do appreciate it, because I'm adding that to my toolbox of things I know unity can now do, but I'm talking about something at the entity level, not scene level

sullen rose Dec 1, 2022, 10:22 PM

#

alright well the best editor for making pinball maps is going to look like unity. i'd suggest using one of the mod making asset store assets that allow your users to author content in the unity editor and share it using an asset bundle

outer mirage Dec 1, 2022, 10:22 PM

#

rotund onyx Hm. Maybe it's due to Unity bug, or your code failed to generate source How is y...

I forgot to say thank you for the recomendation, I will look into filing a bug report, and filtering assembly names (was hoping not to have to do something like that)

sullen rose Dec 1, 2022, 10:23 PM

#

i think if you want to load an xml file and turn it into entities, write C# code that does that. you do not need to make it burst or jobs

#

it isn't performance sensitive

#

it is going to take at most 100ms as plain c# code

#

so it's incomprehensible why you would want to optimize this, this way, by transpiling something (whatever it is) into the subset of C# burstable jobs support

outer mirage Dec 1, 2022, 10:24 PM

#

Not for hundreds of instances of potential components with good user experience, the pinball is more an expression of the system then me making a system for pinball, I just know pinball pretty well

#

Like, I get its acceptable to 'just wait xMS, computers are fast now!' but thats how we get windows 11

sullen rose Dec 1, 2022, 10:25 PM

#

hundreds of instances will be very fast

#

thousands will be fast

outer mirage Dec 1, 2022, 10:25 PM

#

I care about things like a person clicking on icon, and the time it takes with zero hitches, and lots of headroom for everything else I have going on without something like a "Please Wait..."

sullen rose Dec 1, 2022, 10:25 PM

#

i will just say, you aren't the only person who has had to load a save file or a map

#

i don't think burstable jobs will improve that for you

#

i suppose we can have that conversation, but instead i'm hearing stubborn stuff

#

i asked you if you were interested in a better approach. not transpiling to burstable job c# is an important step

#

anyway i gotta go

outer mirage Dec 1, 2022, 10:26 PM

#

Because I've measured the overhead, and until it costs more to set up and run the burst job then the burst job saves, its always worth it paralell and ensure shortest path optimizations

outer mirage Dec 1, 2022, 10:28 PM

#

sullen rose anyway i gotta go

And thanks for talking, regadless of what you think of my design, its good to get feedback, I just wish I had a more succint method to explain the reason I'm doing it this convulted stupid way, when I could just use a hashmap

#Source Generators and asmdef Scoping problems!