Active Effect Migration Exploration

[onyx kernel]

First

[queen kettle]

Second

[onyx kernel]

https://github.com/foundryvtt/dnd5e/issues/2934

[lime perch]

When I run a safe configuration migration there are no errors at all. When I run it a second time after, weirdness ensued lol

somehow it turned the unavailables to passives for no reason, I think it was the original nones turned into passives

[queen kettle]

OK... well... slap that actor on the github issue I suppose? The before and after.

[lime perch]

well this ones actually the after after, its after you do a normal migration and then run the console command after that

[queen kettle]

Still.

[queen kettle]

So when I said "slap that actor on the github issue" I meant the exported json, not just pictures of it

[vast warren]

@onyx kernel @lime perch Do either of you have a world with lots of duplicates and want to test the 3.0.3 build before it is officially released?
https://github.com/foundryvtt/dnd5e/releases/download/release-3.0.3/system.json

[lime perch]

Yep I got a test bed that we were using for midi's scripts I can just rewind it and test for ya, fwiw Tim already released midi in beta form and so far its been pretty clean, the duplicates happen but he has a script that runs to remove the right ones.

[queen kettle]

@fallen hound

[fallen hound]

? , do I need to test anything?

[queen kettle]

Thought you might be interested. 🙂

[fallen hound]

yeah, I reported that bug.

[queen kettle]

Make a backup, I'm sure Jeff would appreesh

[fallen hound]

how should I test it, overwrite the manifest link in system.json, unistall and install from manifest link, or is there a better option? (I have backups, destructive testing is perfectly fine)

[lime perch]

I have a seperate install that has been going back and forth on migration for the last 4 days lol

[queen kettle]

Backing up your world first, then installing directly from github

[lime perch]

I just plopped my live session world into the test bed and hit launch world, then load the backup and repeat

[fallen hound]

okay, restoring the 2.4.latest right now

[lime perch]

make a backup of your world then copy arbron's link above and paste it into this field:

I had wanted to test the migration when you completed it but nobody knew how to compile the system so we kinda figured we'd wait till you did

[fallen hound]

if this works there will be celebrations. I have scheduled a full day of migration and manual fixing with 109 tasks for tomorrow, that would be reduced to like 30 (containers and stuff)

[lime perch]

its probably definitely going to solve the core duplications but tim thought it wouldn't fix the dae/midi ones, we'll see in about 4 minutes when this completes

I really should take out the ddbi compendiums so this test process is faster lol

[fallen hound]

mine is running, too, but will probably take a bit longer -- despite running no automation (GM has was to much stuff in Compendiums...)

[lime perch]

It didn't fix midi/dae duplicates:

no errors at all, I did a safe configuration migration

Looks like it did fix the core dupe bug

Tim has his own script offered in DAE 11.3.3 that can be run and it checks various actors and removes the duplicates, he hasn't solved the issue of ones in compendiums yet cause thats alot more sensitive and most folks have their monsters in a shared compendium, thats on his todo list.

[vast warren]

Weird, it worked perfectly fine for me getting rid of those duplicates when I imported that same actor

[lime perch]

I think when you import a json actor, their ae's become unknown cause the UUID's don't match or something

in dae, you can run a command to fix the imported actors ae's before you migrate, I forge tthe command though

[fallen hound]

sadly have to report, did not work for our world either, and it did duplicate a hand made Lamp timer AE (so that one also nothing DDBimporter), neither Midi nor DAE ever installed for the world

Do you want the pre-migration exported actor?

restoring

the Items the AEs are on are DDBImporter created Items, the AEs are not

[lime perch]

my actors are also ddbi created

[fallen hound]

ddbimporter puts a lot of stuff on the item, but it is all in flags, and for me it does not create the AEs, this world never had DAE, and I created every of the AEs on that Actor by hand. I am not sure I created the Unarmed Defense below by hand btw.

the Unarmored Defense on this one gets duplicated, too (this Actor is a bit less covered in stuff)

also happens with all modules deactivated

[fallen hound]

ah, right, the origin won't match anymore, foundry isn't using (and keeping) GUIDs, right?, the UUIDs are just concatenations of IDs, that are only world-unique and need to be replaced on import, right? Also, they are (abused) as paths (or paths abused as IDs [path meaning entity parent/source - child information]).
all BS, the Items' IDs remain -- now the origin is UUID Item.AE, or...
oh, the origin is UUID starting with Actor, which does change on import

Yeah, so if origin is used testing with exported/imported Actors does not work

[queen kettle]

The embedded documents should be keeping their ids.

(items and effects)

[fallen hound]

but the origin on the AE is a UUID with an Actor not existing after import -- and I think you are using the origin for something in migration

[queen kettle]

The migration script does not look at the actor's id, only the item's id. 🙂

Or specifically, the end of the uuid, so ignoring the Actor.<id> part.

[fallen hound]

ah. okay

[queen kettle]

I found a but

[vast warren]

Good catch, fixed

[queen kettle]

Language! 😱

[vast warren]

Are you sure the migration ran? The de-duplication worked for me on this one :/

[fallen hound]

okay, I will take a look, too, overdue having a look at how dnd5e does their/your/our migration scripts

[fallen hound]

it changes the version information, right -- having a look right now... I mean, it did run migration of course as it duplicated the AE, but I assume you mean if it completed it?
ugh. is there no version information on the Actor? how does it even know what to migrate?

aeh?? he has one effects and two on the sheet?

after migration

and I can delete both

ah, right, Actor.effect does no longer get all effects (which is ...)

restoring for next test...

so the migration does duplicate the Unarmored Defense, what do you @vast warren want me to check -- how do I see what was completed, don't see version numbers on the Actor (well, for ddbi, but don't see dnd5e version number)?

[vast warren]

Well, the migration should be removing the duplicate effects. The version number should be in _stats.systemVersion

[fallen hound]

3.0.3 from the 2.4.max restore with all modules deactivated => duplicate. However, this AE seems to have been ddbi created (the ones on the other character, Gial, are not [Like the Lamp timer]). Besides, with all modules deactivated it should not matter, right?

[vast warren]

So if I (a) load 3.0.2, (b) create an actor and import the "Ketris Bearsmasher" information, (c) leave world, update to 3.0.3, (d) enter world, wait for migrations to run, then it removes the duplicate effects properly for me:

(before / after)

[fallen hound]

then, is there any branching in the migration script (of 3.0.3) that would be sensitive to starting version 2.4.max/3.0.2?

[vast warren]

Nope, it should run on any actor that hasn't yet been updated for 3.0.3

Hmm, let me check something

[fallen hound]

are you aware of anyone not using ddbi having the problem? I mean, should not matter with everything deactivated, but at least one would have a good indication where to look in the data structure/ how it is read by the migration script

I'll have a look into the migration.mjs now, too, there is only one migration script in dnd5e, right?

[vast warren]

That's the only one

[queen kettle]

🤨

There is no way this should migrate correctly at all, one of them doesnt even have an origin

[vast warren]

What actor is that data from?

[queen kettle]

Bearsmasher up there

[vast warren]

Hmm, looking at the raw JSON it definately does have an origin. In fact that is what that diff is saying, that one has origin and the other doesn't, otherwise it wouldn't appear in the diff

[queen kettle]

The one on the item does not have an origin

[vast warren]

Correct, because it is the origin

We only care about the origin of effects on the actor, to match them back to their original effect on the item

[fallen hound]

okay, so you found the ddbi related bug then (seems the bearsmasher AE was created by ddbi [despite not having DAE]) guess not if #1207781431011442740 message
true, #1207781431011442740 message, so I again believe that might be the ddbi part. but what about Gial, I made all those AEs by hand?

[queen kettle]

Ah - the origins do not have to match, it looks at the item id only

[vast warren]

¯\_(ツ)_/¯, Bearsmasher migrates perfectly for me, so I'm not sure where the problem is coming from

I did do another build of 3.0.3 which may or may not fix anything at all

[queen kettle]

wonder if something is getting changed during the export that muddies this 🤔

[vast warren]

To fix that, I'll just give you my mailing address and you can send your entire computer over to me for testing, nothing could be different then 🤪

[fallen hound]

Sadly the world is 5+GB and has tons of assets the GM, not I, owns. I did think about stripping it down to Gial, but...

[vast warren]

Just tested an imported Gial, also migrated properly

Can you try creating a fresh world and importing your own JSON to test? Make sure you create the world and import in 3.0.2 or earlier

[fallen hound]

on my way. sleep is for the weak (I am in Germany), but seriously, as long as someone tries to solve my bug, I am at it -- restoring...

[queen kettle]

youre gonna pay for shipping in this economy?

[fallen hound]

should write a feature request to add the systemversion to the export

[vast warren]

It is in exported data, down at the bottom in the _stats block

[fallen hound]

... and I am blind, thx (well, to be fair, I am testing and documenting todos for migration tomorrow for 12 hours now)

[vast warren]

It is also added to a flag, flags.exportSource.systemVersion

[fallen hound]

Hmmm, does indeed work. So an export/import does fix whatever in the data (has to be data and dnd5e/core only as all modules are of for the whole test) lets the migration duplicate/fail to deduplicate
2.4.1 => 3.0.3

without wanting to shoot myself into the foot: how many users are reporting problems? If I have to do the 109 tasks for migration I'll be pissed, but won't stop using FVTT

[queen kettle]

🤔

is this where we run a big, fat, diffObject on the actors, the one that failed and the one that succeeded

or rather, you do, since the "fix" (?) happens either during export or import 🤔 assuming it's not related to the world

[fallen hound]

okay, so I need to import it into the original world (and I guess I'll sufix its name to not get completely confused), right? That in 2.4.1, and then migrate and hope one works and one bugs, right? And if so look at the objets in 2.4.1 on console?

[queen kettle]

Not really sure. 🙂

[vast warren]

I'd say import in 2.4.1 or 3.0.2 and do a diff before migration foundry.utils.diffObject(actorA.toObject(), actorB.toObject())

[fallen hound]

on my way

just getting some coffein before that

[queen kettle]

$5 says it's something stupid and the fix is 1 line of code. always is.

[fallen hound]

like the Arcane Recovery rewrite btw

[queen kettle]

Thanks!

[fallen hound]

should I rename one now, or...

[vast warren]

Sure, rename and try the migration, then I guess we'll look at an after diff

[fallen hound]

this will take a while -- empty Gial world migration is somewhat faster than real world oversized with assets

when did you @vast warren release the 2nd 3.0.3 version, and what did you change? both gials are okay! (ketris, too, all seems fine)

[queen kettle]

50 mins ago

[vast warren]

Yey

[fallen hound]

okay, that might have been after the last full cycle before this. what did you @vast warren change?

... why don't I just look at the commit..

which branch is it?

[vast warren]

I modified slightly how it checks version number to ensure the migration ran correctly

This commit: https://github.com/foundryvtt/dnd5e/commit/a146c32a195c1164cf0bf36292de2bffa4bfebc4

[fallen hound]

okay, thanks a lot -- tomorrow only adopting new features related migration work, not fixing bugged ones. 🥳

[eternal robin]

I tested 3.0.3 and it fixes most effect duplications. There are actors , however whose effect origin is broken (if they were imported from another world and so on) that this will not fix but I think it's the best you can do and fixes most of the cases I can see.

[vast warren]

Don’t want to get too aggressive and end up deleting effects we shouldn’t have. Thanks for testing

[eternal robin]

It would be nice if for transfer effects you set the origin to the item uuid. That way on the legacy sheet the source won't be unknown.

[eternal robin]

I did notice that for unlinked tokens with effects to remove that the following error is thrown:
actor.mjs:3124 Uncaught (in promise) TypeError: Cannot read properties of null (reading 'visible')
at Actor5e._displayTokenEffect (actor.mjs:3124:26)
at Actor5e._onUpdate (actor.mjs:3079:10)
at ActorDelta._onUpdate (foundry.js:18542:25)
at ClientDatabaseBackend.callback (foundry.js:13738:11)
at foundry.js:13712:43
at Array.map (<anonymous>)
at #handleUpdateDocuments (foundry.js:13712:33)
at async Actor5e.updateDocuments (commons.js:8001:23)
at async Actor5e.update (commons.js:8098:23)
at async migrateWorld (migration.mjs:119:11)

[eternal robin]

I noticed the problem - was just creating a comment when I saw the updated commit.

[lime perch]

Is there a way to get a script that runs the migration on a module compendium? A very large group of users use Dndbeyond importer and they typically will have the monsters in a shared compendium, and it doesn't get touched by the migration right?

[vast warren]

dnd5e.migrations.migrateCompendium(game.packs.get("my.compendium"));