#JSON Deserialization: Resolving Catalog Objects given a String Key

1 messages · Page 1 of 1 (latest)

vague light
#

This is a rather complex problem I've been stuck on for a while.

I'm building a solution where I deserialize content (e.g., level configs, entities) from JSON payloads using the new com.unity.serialization.json package. Each JSON object includes a "Definition" node (a string key) that needs to be indexed into a runtime catalog for lookups. To ensure reliable resolution (e.g., turning a key like "ProjectilePrefab" into a GameObject), the lookup must happen after all the objects are deserialized and registered in the catalog, as doing it mid-deserial (via adapters) risks "half-done" results, where someone might attempt to reference something that has not been created yet, especially for staggered dependencies where the catalog sources are in the same JSON.

The challenge escalates with cross-object references: For example, in a struct like this:

public struct EntityConfig : IComponentData
{
    public Entity EntityPrefab;  // To be populated via catalog lookup on Definition
    public EntityConfig NestedEntityConfig;  // To be populated via cross-ref to another EntityConfig's instance
}

Dictionary<string, EntityConfig> KeysToEntityConfigs = ...


[
  {
     "Definition": "FirstDef",
     "Type": "EntityConfig",
     "Data": {
        "EntityPrefab": "SomeEntity"
        "NestedEntityConfig": "SecondDef"
      },
  },
  {
     "Definition": "SecondDef",
     "Type": "EntityConfig",
     "Data": {
        "EntityPrefab": "..."
        "NestedEntityConfig": "..."
      },
   }  
]
#

I find it impossible to resolve this JSON, as, if for example I'm using Adapters, even if I define a custom adapter for the Entity type, when Visitation occurs on that node, the Def its trying to visit won't exist yet, and thus is unable to resolve. by the time everything is resolved, and everything exists, I cannot use JSON tools as I'm already outside that environment.

If I define an Adapter for the struct, it might work for the first level, but once its nested somewhere (as its in this example) the Adapter will expect the same json structure, instead of being provided with a string which is meant to look up somewhere as with the Entity.

I'm been stuck on this for a couple of weeks by now and I have no idea how to progress, latest Idea I've had was marking the target fields for custom catalog deserialization with Attributes:

public struct EntityConfig : IComponentData
{
    [NeedsDefCatalogResolve]
    public Entity EntityPrefab;  // To be populated via catalog lookup on Definition
    [NeedsDefCatalogResolve]    
    public EntityConfig NestedEntityConfig;  // To be populated via cross-ref to another EntityConfig's instance
}

This allows me to visit the object with Unity.Properties and register them somewhere for later resolve, but I don't know enough Unity.Properties and visitors to be able to pass the string key from the JSON into this. From what I am aware, even if I create a Visitor, that's after being deserialized, and I have no access to the JSON data there unlike how Adapters do.

#

This is a rather complex subject and Im not sure how to properly explain it, but I hope the pseudocode helps paint an image

sharp kindle
#

I'll start by saying that I think I may be misunderstanding some part of the problem, but here goes

why are you forcing yourself to stay in json tools? I don't think there's anything wrong with processing the data into an intermediate format first (i.e. to defer entity creation)

if you really want to do this in that way, if you have control over the creation of the json, you could sort it in the order of dependencies on creation (assuming the library will then process it in that order as well)

#

also isn't there inherently a problem with the data structure in your example, which is self-referencing?

vague light
#

This kinda comes from before I found out about Content Archives and the Entities content management, from there I justify it as a somewhat size-fits-all silver bullet to create (and iterate) content easily without having access to the Editor. It's human readable as well and that's why I chose it instead of other formats. The fact that Content Archives cannot be disposed of at runtime, while I can load a JSON with some content and then destroy what they created is another bonus. The alternative to this would've been Scriptable Objects.

Also I am creating it in the order of dependencies, and at first it didn't have this two-step deferred creation, but for the sake of flexibility and less errors I decided it would be the best to have it that way instead of every single content in each json having to be carefully placed to be able to be referenced.

Regarding self-referencing, so far I just have a key (definition) check, and regarding structs I dont think that would be an issue given that they are value types

#

Since I posted this thread I thought about other solutions to the problem, for example the issue that the Json Deserializer ignores fields that do not match, I can still fetch them myself manually while its still being deserialized, if I create a visitor that gets a struct passed in to hold some data to then take out of the visitor and into my own context, but I need to get around it

#

Kinda down a rabbit hole where if I cannot get out i'll just do simple entity and data creation and leave everything else to the Editor

sharp kindle
#

regarding structs I dont think that would be an issue given that they are value types
well, the struct you gave adds a field to its own type in its definition, that's not possible since it would be a cyclic struct layout, so I assume you did something else?

#

if it was a key to look up an entity config it would make more sense (like in your json)

vague light
#

Forgive me for the lack of response as work kinda sucks me in and out, I'll try reviewing it this weekend, alongside with the new Content Management samples that Unity has put out

vague light
sharp kindle
summer bane
#

If it's any help, heres how I'm doing my own json deserialization code

{
  "Label" : "MyDiagram",
  "Layers" : [
    {
      "Label" : "MyLayer",
      "Nodes" : [
        {
          "EffectType" : "Slope",
          "Mode" : "Set"
        }
      ]
    }
  ]
}```Which matches these 3 classes:
```cs
public class DiagramDTO {
  [JsonProperty("Label")] public string name;
  [JsonProperty("Layers")] public List<LayerDTO> layers = new();
}
public class LayerDTO {
  [JsonProperty("Label")] public string name;
  [JsonProperty("Nodes")] public List<NodeDTO> nodes = new();
}
public class NodeDTO {
  [JsonProperty("EffectType")] public EffectType type;
  [JsonProperty("Mode")] public EffectMode mode;
}```Deserialising is straight forward:
```cs
public class VoroDiagram : MonoBehaviour {
  void LoadDiagram() {
    var jsonSource = Resources.Load<TextAsset>("Template"); // Template.json
    var root = JObject.Parse(jsonSource.text);
    var diagramDto = root.ToObject<DiagramDTO>();
  }
}```
sharp kindle
#

If you care about performance, e.g. avoiding allocations & managed code, there may be a reason to use the unity package (though I haven't compared them, I expect the unity package to be more performant, but I may be wrong)

vague light
#

The unity package does use unmanaged structs, instead of pooling like System.Text.Json

vague light
# sharp kindle i mean, your struct layout should look like that as well, no?

I want to deserialize directly into the structs as-is, and at runtime the key (or any other json data) is essentially garbage data.

The first time I implemented this logic and the first working version I had, I made a "middle step struct / class" that did match the struct and then through an Interface would convert that data into the real final data as each one of these classes implemented its own logic to transform themselves into the useful runtime data. Got rid of it as that's insane to work with every single struct and class in the project

sharp kindle
#

I'm not sure I'm on the right track, but as I understand it the main challenge is to automate the patching of the references between the items?

Do you do the parsing and reference patch-ups in two separate steps?

To not have to maintain parsing structs for every type, you could mark the various referenced types (e.g. entity config reference, entity reference, etc) with attributes

e.g. [JsonReferenced(typeof(EntityConfig))]

then at runtime keep a mapping of something like <string, EntityConfig> like you already showed, and just go over all the data that have these attributes to replace it with the data you want

#

you could even use code-gen to make this a bit more efficient at runtime, so you don't have to use reflection, but that could be a later step (if/when you see performance is an issue)

vague light
#

For now I'm leveraging Unity Serialization, which uses Unity Properties which automatically uses codegen to avoid using reflection, or uses reflection if the code wasn't generated

# 
    [AttributeUsage(AttributeTargets.Field)]
    public class DefReferenceTargetAttribute: Attribute
    {
        public FixedString32Bytes JsonNodeContainingDefName { get => AttributeData.JsonNodeContainingDefName; } 
        public FixedString32Bytes FieldName { get => AttributeData.FieldNameToUse; } 
        public DefReferenceTargetAttributeData AttributeData;
        public DefReferenceTargetAttribute(string jsonNodeContainingDefName, string fieldName = null)
        {
            // Sanity checks...
            
            AttributeData = new DefReferenceTargetAttributeData
            {
                JsonNodeContainingDefName = jsonNodeContainingDefName,
                FieldNameToUse = fieldName
            };
        }
    }
    
    public struct DefReferenceTargetAttributeData : IEquatable<DefReferenceTargetAttributeData>
    {
        public FixedString32Bytes JsonNodeContainingDefName;
        public FixedString32Bytes FieldNameToUse;
# 
    class ReferenceResolutionLowLevelVisitor : IPropertyBagVisitor, IPropertyVisitor
    {
        public ContentLoadDeserializationContext Context { get; set; }
        public ContentPackId PackId { get; set; }
        public List<(DefReferenceTargetPropertyData, DefReferenceTargetAttributeData)> PropertyAndAttributeData;

        void IPropertyBagVisitor.Visit<TContainer>(IPropertyBag<TContainer> propertyBag, ref TContainer container)
        {
            foreach (var property in propertyBag.GetProperties(ref container))
            {
                property.Accept(this, ref container);
            }
        }
        void IPropertyVisitor.Visit<TContainer, TValue>(Property<TContainer, TValue> property, ref TContainer container)
        {
            var value = property.GetValue(ref container);
            var defReferenceTargetAttribute = property.GetAttribute<DefReferenceTargetAttribute>();
            if (defReferenceTargetAttribute != null && value != null)
            {
                if (!defReferenceTargetAttribute.JsonNodeContainingDefName.IsEmpty) // This should never happen due to restrictions in the Attribute cctor.
                {
                    var data = new DefReferenceTargetPropertyData()
                    {
                        Property = property,
                        TypeHash = CachedTypeRegistry.GetStableTypeHash<TValue>()
                    };
                    PropertyAndAttributeData?.Add((data, defReferenceTargetAttribute.AttributeData));
                }
            }
            // Recurse if container type
            if (TypeTraits.IsContainer(typeof(TValue)) && value != null)
            {
                PropertyContainer.Accept(this, ref value);
            }
        }
    }
    
    public struct DefReferenceTargetPropertyData : IEquatable<DefReferenceTargetPropertyData>
    {
        public IProperty Property;
        public ulong TypeHash; 
    }
#

Then this visitor gets used to deserialize the additional json nodes for a deferred two pass resolution:

                // Collect properties that requested deferred resolution via visitor. The visitor will add entries to
                // both the local list (propertiesWithReferencesList) and the context.PendingReferences list.
                var propertyAndAttributeData = new List<(DefReferenceTargetPropertyData, DefReferenceTargetAttributeData)>();
                var visitor = new ReferenceResolutionLowLevelVisitor
                {
                    Context = _deserializationContext,
                    PackId = _packId,
                    PropertyAndAttributeData = propertyAndAttributeData
                };
                PropertyContainer.Accept(visitor, ref thingData);

                foreach (var (propertyData, attributeData) in propertyAndAttributeData)
                {
                    thingObj.TryGetValue(attributeData.JsonNodeContainingDefName, out var view);
                    if (view.IsNull())
                        continue;

                    var pendingReference = new PendingThingReference()
                    {
                        AttributeData = attributeData,
                        PropertyData = propertyData,
                        RequestingThingTypeHash = typeInfo.StableTypeHash,
                        RequestingDefId = _defId,
                        WantedDefId = new DefId(view.AsFixedString<FixedString64Bytes>())
                    };
                    
                    _deserializationContext.PendingReferences.Add(pendingReference);
                }
#

PendingThingReference being:


    /// <summary>
    /// Pending references for post-resolution.
    /// Created by <see cref="ReferenceResolutionLowLevelVisitor"/> when visiting a Thing instance whose Type contains fields with <see cref="DefReferenceTargetAttribute"/>.
    /// </summary>
    public struct PendingThingReference : IEquatable<PendingThingReference>
    {
        /// <summary>
        /// Unique by restriction, as it'd be confusing if multiple fields had the same <see cref="DefReferenceTargetAttributeData"/> definition.
        /// </summary>
        /// <remarks>Uniqueness does not apply if the requesting Type is a buffer, as there can be multiple instances of the same Type requesting.</remarks>
        public DefReferenceTargetAttributeData AttributeData;
        
        /// <summary>
        /// Unique, as a single object will not have duplicated <see cref="IProperty"/>.
        /// </summary>
        /// <remarks>Uniqueness does not apply if the requesting Type is a buffer, as there can be multiple instances of the same Type requesting.</remarks>
        public DefReferenceTargetPropertyData PropertyData;
        
        /// <summary>
        /// Pack Id that is requesting the DefId.
        /// </summary>
        public ContentPackId RequestingPackId;
        
        /// <summary>
        /// DefId of the Thing instance that is requesting the DefId.
        /// </summary>
        public DefId RequestingDefId;
        
        /// <summary>
        /// Type Hash of the Thing instance that is requesting the DefId.
        /// Duplicated Hashes can exist, as one single object can request multiple different references.
        /// </summary>
        public ulong RequestingThingTypeHash;
        
        /// <summary>
        /// DefId to resolve.
        /// </summary>
        public DefId WantedDefId;
    }
#

Unity Properties is what Unity internally uses for tools such as visual scripting / graphs, as far as I am aware

#

And Unity Serialization uses it as well

#

It's 100% a step up from my previous system, right now I'm trying to figure out how to solve buffer / list json structures, where there can be multiple instances of the same Type

#

as otherwise this would be a 1:1 relationship given the RequestingDefId and RequestingThingTypeHash

#

Well not really as multiple requests can come out of the same type, 1:1:many relationship then?

#

Also this is pretty much untested code so I'm unsure if it even works, for example the Visitor I am unaware of how it traverses the container and how the properties are read

#

It'll be matter of just tweaking it a bit if that happens

sharp kindle
#

ah okay, great, I think you're well under way

#

seems we came to the same conclusion

sharp kindle
#

im not familiar with these APIs unfortunately, but I think you'll get there 😄