Trying to save a TMap with a Instanced UObject derived class

[wicked sapphire]

hi again guys, I've been stuck trying to implement saving, but I'm doing something wrong.

I have this:

UCLASS(BlueprintType, Blueprintable)
class TJCONDITIONS_API UTJConditionMap : public UObject
{
    GENERATED_BODY()

public:
    UFUNCTION(BlueprintCallable)
    void GetSaveData(FTJConditionMapSaveData& SaveData); 

    UFUNCTION(BlueprintCallable)
    void LoadFromSaveData(const FTJConditionMapSaveData& SaveData);

public:

    UPROPERTY(EditAnywhere, Instanced, BlueprintReadOnly, SaveGame)
    TMap<FName, TObjectPtr<UTJCondition>> ConditionMap;

    UPROPERTY(BlueprintAssignable)
    FOnConditionFromMapMet OnConditionFromMapMet;
    UPROPERTY(BlueprintAssignable)
    FOnConditionFromMapChangedState OnConditionFromMapChangedState;  
};

My intention is to save the ConditionMap.

the structure for saving the ConditionMap is:

USTRUCT(BlueprintType)
struct FTJConditionMapSaveData
{
    GENERATED_BODY()

public:
    UPROPERTY()
    TSubclassOf<UTJConditionMap> ActualClass;
    UPROPERTY()
    TArray<uint8> ByteData;
};

and the implementation of the functions that use it are:

void UTJConditionMap::GetSaveData(FTJConditionMapSaveData& SaveData)
{
    SaveData.ActualClass = GetClass();
    FMemoryWriter MemWriter(SaveData.ByteData);
    
    FObjectAndNameAsStringProxyArchive Ar(MemWriter, true);
    Ar.ArIsSaveGame = true;
    Serialize(Ar);
}

void UTJConditionMap::LoadFromSaveData(const FTJConditionMapSaveData& SaveData)
{
    FMemoryReader PropMemReader(SaveData.ByteData);
    FObjectAndNameAsStringProxyArchive PropAr(PropMemReader, true);
    PropAr.ArIsSaveGame = true;
    Serialize(PropAr);
}

any idea why the TMap<FName, TObjectPtr<UTJCondition>> is not getting saved/loaded correctly?
the map gets saved and loaded, but the conditions dissapear.

[marsh cypress]

Instanced objects don't get serialized the way you want them to when using a default archiver like that.
The call to Serialize will only serialize the reference to the other object, not the object itself.
I've only seen this to work (and made it work myself) by creating a custom archiver.

There was a short conversation about the same thing yesterday.

[wicked sapphire]

is there any documentation regarding how to create a custom archiver?

[marsh cypress]

Not that I know of, but I never looked for it either.
I don't remember how much of that this talk covered
https://dev.epicgames.com/community/learning/talks-and-demos/4ORW/unreal-engine-serialization-best-practices-and-techniques

[wicked sapphire]

thanks for the link! Any tips or suggestions about how to go about this, or what considerations to have?

[marsh cypress]

Not off-hand. My archiver is based on serializing everything for the save game (because that's my use case for this stuff too) in one go.
Not converting individual objects to structs which are on the save game.
So it's much more complicated to handle many more cases.
Like mine will preserve asset references and cross object references (like Object A with a property pointing at Object B) as long as both objects are included in the save.

I'm hoping to have it included in a plugin at some point in the future so that I can link to it in cases like this, but I don't have a timeline for that yet.

[wicked sapphire]

okey, I'm going to watch the talk and think about it, I'll keep posting here with new developments.
Thanks for your help 🙂

[marsh cypress]

Adding @frigid patio 'cause he was the one I had this same conversation with yesterday. And if I get it posted (which I might be able to do even with the functionality I think is missing) I can drop a link here for both of you.

[frigid patio]

thanks, but i got it sorted out for me with custom object de/serialization

[wicked sapphire]

could you expand on this? im guessing you are talking about overriding the Serialize() function?

[frigid patio]

no, i use my own save system. I had trouble with instanced objects and now i just iterate over them, serialize each on it's own and reinstanciate them when deserializing

[wicked sapphire]

Hi! I've been able to solve my issue, so I'll write how I did it and close the thread.
I'll provide as simple an explanation for the problem I had and how I solved it, in case someone can benefit from it later.

The problem I had is related to saving UObject derived classes with the Instanced UPROPERTY specifier, like so:

UPROPERTY(Instanced)
TObjectPtr<UInstanceExample> Example.

The real issue was that I had Instanced Objects that had other instanced objects inside, and Unreal by default doesn't save UObject*, even if they are marked with Instanced.
To make this work, without having to dive too deeply into Unreal's save system, FArchive, and Serialize workflow (which might be a better alternative, as Ramius suggested), I did the following things.

First, in the UInstanceExample, which we'll say it the base class for all instanced classes, we add :

UPROPERTY()
FGuid InstanceID;

When the InstanceExample is created, it fills that ID using FGuid::NewGuid().

Second, I created a USTRUCT() that would then be held in the USaveGame* derived class, following regular Unreal Save System functionality.

The struct is quite simple, and looks something like this:

USTRUCT(BlueprintType)
struct FExampleSaveData
{
    GENERATED_BODY()

public:
    UPROPERTY()
    TSubclassOf<UInstanceExample> ActualClass;
    UPROPERTY()
    TArray<uint8> ByteData;
    UPROPERTY()
    TArray<FGuid> ChildConditions; 
};

We store the actual class of the UObject, filling it with GetClass() when saving and using NewObject<UInstanceExample>(ExampleOuter, ActualClass) when loading.
We store the ByteData, so we can use Serialize() to both save and load.
And the interesting part is the ChildConnections array, which stores the ID of the nested instanced objects it had.
When saving, the UInstanceExample class gets it's nested objects and stores their ID.
When loading, said ID will be used to retrieve the loaded instance, and do what we want with it.

The third step is quite tangled with the design of my system, so it probably won't work for you.
In my case, my UInstanceExample are always held in a class, that we'll call InstanceMap.
The InstanceMap class, as it name implies, has a :

UPROPERTY(Instanced)
TMap<FName, TObjectPtr<UInstanceExample>> Map.

In this map are all the InstanceExample we want to save.
The map has a GetSaveData and LoadFromSaveData functions.
For simplicity, we'll say that the GetSaveData just returns a array of our FExampleSaveData,
and the LoadFromSaveData takes an Array of FExampleSaveData.

Another thing we add to the UInstanceExample class is a method like this one:

virtual void GetSaveData(FExampleSaveData& ExSaveData)

The base class implements it like this:

void UInstanceExample::GetSaveData(FExampleSaveData& SaveData)
{
    SaveData.ActualClass = GetClass(); 
    FMemoryWriter ConMemWriter(SaveData.ByteData);
    FObjectAndNameAsStringProxyArchive ConAr(ConMemWriter, true);
    ConAr.ArIsSaveGame = true;
    // Serialize(ConAr);
    // Explicitly serialize UPROPERTY marked with SaveGame
    GetClass()->SerializeTaggedProperties(ConAr, (uint8*)this, GetClass(), nullptr);
}

(I use SerializeTaggedProperties instead of Serialize directly because Serialize just wasn't working for some reason 😦 )

Derived classes from UInstanceExample that actually contain other nested conditions populate the SaveData.ChildConditions array using their nested conditions.

[marsh cypress]

Yup, that is the very manual way to serialize sub objects that you’ve got to do if you don’t use the archiver strategy.
Congrats on getting it working!

[wicked sapphire]

Now we go back to the InstanceMap, and it's GetSaveData.
The implementation would look something like this:

UInstanceMap::GetSaveData(TArray<FExampleSaveData>& ArrayOfSaveData)
{
  TArray<UInstanceExample> AlreadySaved;
  //First, loop through our Map
  for(const TTuple<FName, TObjectPtr<UInstanceExample>>& Element : Map)
  {
    FExampleSaveData SaveData;
    //Call GetSaveData from our UInstanceExample object
    Element.Value->GetSaveData(SaveData);
    ArrayOfSaveData.Add(SaveData);
    //We will see why later
    AlreadySaved.AddUnique(Element.Value);
  }
  
  //After this, we have all the base level InstanceExample objects, 
  //but we still need to get their nested objects. How?
  //What I do is have a method that iterates through all the UInstanceExample, and sees
  //if they have the InstanceMap in their Outer chain, basically meaning they are indeed
  //inside the Map. If they are, it adds them to an array that it then returns.
  TArray<UInstanceExample> AllInstances = FindConditionsInOuterChain(this);
  
  //Then we do a final loop through AllInstances like so:
  for (UInstanceExample* Instance: AllContainedConditions)
  {
    //Already saved, meaning it was directly in the Map
    if (AlreadySavedConditions.Contains(Condition)) continue;

    FExampleSaveData SaveData;
    Instance->GetSaveData(SaveData);
    ArrayOfSaveData.Add(SaveData);
    AlreadySaved.AddUnique(Condition);
  }     
  
  //And that's it, we have saved all our conditions, even the nested ones, and the connections
  //between them    
}

[wicked sapphire]

thanks! After seeing the talk you sent me, I could see more or less how I could go about doing it that way, but I'm determined to finish the actual code this month for the plugin, so I didn't want to have to dive deep into FArchive, etc...

After all, I can always go back and save it that way after it's released.
(let's be honest, I probably won't if no one complains :D)

[wicked sapphire]

After this, the LoadFromSaveData() should be quite straightforward to imagine, so I won't write it here.
Instead I'll give a summary of what it does.

{
  //Go through the incoming array of FExampleSaveData, that contains the actual save data.
  
  //For each element, create an actual UInstanceExample, using the saved type, 
  //load it's properties using the saved ByteArray

  //After we've created the UInstanceExample with their actual saved data,
  //we are ready to resolve their connections.
  //I'm too lazy to say how, but I'll say I used an interface to know which
  //instances have nested objects, and they implement a interface method that resolves it
  
}

And just like that, we save and load instanced objects with nested objects inside.

[frigid patio]

i feel like you would have a easier time when you would serialize instances WITH their nested instances

so the object would implement something like De/Serialize() and serialize itself together with it's instances

if you call that recursive you can have infinite nesting without the FGuid lookup

[wicked sapphire]

@marsh cypress do you think this is a fine approach? can it be a problem to have to iterate twice when loading?
For my use case, I expect at most thousands of items, and that would be in big projects.

[wicked sapphire]

you have a very valid point, and it was my original thought, but since I'm saving using USTRUCT()s, I couldn't figure it out

[wicked sapphire]

could you expand a bit more?

how exactly would you save something like the class for example?

[frigid patio]

bool UDataAsset_RoomDefinition::GameStateObject_ToJSON_Object(TSharedPtr<FJsonObject>& JsonObject) const
{
    TArray<TSharedPtr<FJsonValue>> DataArray;    
    for(UDataAsset_RoomDefinition *Room : Rooms)
    {
        TSharedPtr<FJsonObject> Entry = MakeShared<FJsonObject>();
        USubSystem_GameState::SerializeObject(Room, Entry);  // this pretty much serializes the object binary and calls the interface method GameStateObject_ToJSON_Object() on the subobject
        DataArray.Add(MakeShared<FJsonValueObject>(FJsonValueObject(Entry)));
    }

    JsonObject = MakeShared<FJsonObject>();
    JsonObject->SetArrayField("data", DataArray);
    return true;
}

now in my case it's json, but that json array could also be an normal array on an object/struct

[wicked sapphire]

I kind of understand. I'm curious about one thing though.
Does the FJsonObject, have an array of FJsonValue, and the
FJsonValue is the parent class of the FJSonValueObject?

[frigid patio]

yea, those are unreal structs for handling json stuff

[wicked sapphire]

so you end up with a single JsonObject, that has a list of FJsonValues with their saved values?

[frigid patio]

i end up with smth like this

{
    "data": [
        {
            "data": [
                {
                    "data": [
                        {
                            "data": [],
                            "objectData": "serializedBinaryBlob"
                        },
                        {
                            "data": [],
                            "objectData": "serializedBinaryBlob"
                        }],
                    "objectData": "serializedBinaryBlob"
                },
                {
                    "data": [],
                    "objectData": "serializedBinaryBlob"
                },
                {
                    "data": [],
                    "objectData": "serializedBinaryBlob"
                },
            ],
            "objectData": "serializedBinaryBlob"
        }
    ],
    "objectData": "serializedBinaryBlob"
}

"data" holds my custom serialized fields, and objectData keeps binary blobs from unreal serialization

and when i deserialize it unfolds from the root and the objects deserialize themselves with their instanced objects (which also deserialize themself....)

[wicked sapphire]

i see, that is quite cool tbh

btw, listening to the talk Ramius sent yesterday, it said something about saving with JSON not working in builds. I don't know if it's relevant for you, but you might want to look it up

[frigid patio]

that may be related to the json stuff from unreal serialization

there's an option to output the binary stuff as json, too

[wicked sapphire]

I think the guy said they exclude Json save functionality in packaged builds, since they want binary option

[frigid patio]

but afaik no way to import/load that json

ah yea i think i saw that talk, they iirc did the right thing to use UE serialization instead of cooking their own soup 😄

[marsh cypress]

Yeah, it's perfectly serviceable. So unless you run into specific problems it's probably good enough.

1. You should be careful about what sort of saving processes you change after you release. The last thing you want to do is invalidate player's saves (maybe that's okay if it's a beta or something and you're not supposed to carry over any progress). So any large structural changes to saves should happen before then and only require minor tweaks after release.
2. two iterations shouldn't be a problem. Even though I do mine with an archiver, I still do two passes to do the saving or loading. To save pass one figures out all the objects & subobjects that should be saved and pass two saves them out. On load, pass one instantiates all the objects and then pass two does the call to Serialize on each of them.

[wicked sapphire]

nice, thanks for your time!
even though we use different things, our passes are quite similar ahhaha