V14 DataFieldOperator Replacement of Special Operation Keys

[wheat wraith]

Hi friends. One of our first V14 epics is complete. This is an internal API change which eliminates special string operation keys for DataModel#updateSource and Document#update workflows.

A brief overview of the change is below, with more technical details and related issues mentioned in the following GitHub issue:

https://github.com/foundryvtt/foundryvtt/issues/13090

Please use this thread for any questions you have about how this upcoming change works or how it will affect your systems and modules.

Overview of Change

This change eliminates special database operation keys like {==key: replacement} or {-=key: null} in favor of explicit special operations which share the actual field name. Specifically:

- {-=key: null}
+ {key: ForcedDeletion.create()}

- {==key: replacement}
+ {key: ForcedReplacement.create(replacement)}

The paramount advantage of this is that special operation keys share the same key as regular operations, and we avoid situations where a database update might contain something like {key: "foo", "-=key": null, "==key": "bar"} leading to significant ambiguity in what actual operation will be performed. This issue also manifests in downstream code where understanding what fields changed requires you to test things like:

if ( ("key" in data) || ("==key" in data) || ("-=key" in data) ) {
  ...
}

This change is labeled as BREAKING because it changes the way that developers should interact with DataModel#updateSource and Document#update APIs. It is not immediately breaking, as a deprecation period preserves full compatibility for special deletion and replacement keys until Version 16.

GlobalThis References

We have added these globalThis shorthand references to make it more convenient to interact with these special operators:

/**
 * A singleton ForcedDeletion operator instance that can be reused.
 * @type {foundry.data.operators.ForcedDeletion}
 */
globalThis._del = new foundry.data.operators.ForcedDeletion();

/**
 * A reference for ForcedReplacement.create that can be easily referenced.
 * @type {foundry.data.operators.ForcedReplacement.create}
 */
globalThis._replace = foundry.data.operators.ForcedReplacement.create;

Checking Whether You Are Impacted

The following regex pattern is a good one to use to look for possible usages of a legacy forced deletion (-=) or forced replacement (==) key:

["'`\.][=-]=

Example Usages

Example 1 - Reset a field back to its initial value

model.updateSource({someField: foundry.data.operators.ForcedDeletion.create()});

doc.update({"system.attributes.strength.value": _del}};

Example 2 - Forcibly replace a certain value

model.updateSource({someField: foundry.data.operators.ForcedReplacement.create("foo")});

doc.update({"system.attributes.strength.value": _replace(17)}};

Example 3 - Forced Replacement of an entire object

model.updateSource({innerSchema: foundry.data.operators.ForcedReplacement.create({foo: "bar"})});

doc.update({"system.attributes.strength": _replace({value: 17, bonus: 2)}};

V14 DataFieldOperator Replacement of Special Operation Keys

[shell escarp]

I... guess I like the change, but I am afraid of how much this breaks.

[river fulcrum]

Might take some getting used to but in general I like the change.

Will the old syntax be immedeatly removed without any grace period?

[silk sundial]

Deprecation 'til v16

[river fulcrum]

reading helps 😄

[finite gazelle]

It's better, but I find the new Forced deletion() just a bit too clunky

[wheat wraith]

Perhaps there is a more clear statement on compatibility that could be added to the documented issue? Do you have specific concerns, or generally vague bad vibes?

[wheat wraith]

Understandable, but thankfully such operators are pretty rarely used (at least in my experience)

[forest fractal]

If anything there's always making a factory out of it, I suppose

f: () => new ForcedDeletion()

Could be an included utility

[shell escarp]

Vague vibes. That syntax has been very integral and difficult to teach, and I feel the new one is even harder, especially for the casual macro polo user.

[narrow sky]

That's gonna take some getting used to lol
We use forced deletion / replacement a lot, especially modules that have to rely entirely on flags, such as CCM.
100% gonna make a wrapper for that

[wheat wraith]

yeah, perhaps so. Unfortunately the technical debt from incorporating the operator into the key was unsustainable.

And a huge source of bugs that are likely affecting your systems/modules without you really realizing it.

[formal basalt]

those ```js
if ( ("key" in data) || ("==key" in data) || ("-=key" in data) ) {

[wheat wraith]

Way too much downstream code dependent on checks like if ( "x" in diff ) { ... } or if ( change.x === "foo" ) ...

[idle grove]

There is unsetFlag at least, unless you had objects instead of a simple primitive as a value.

[forest fractal]

Lots and lots of objects for modules like these...

[topaz hornet]

Yeah, the syntax of "pass this magic function as a value" feels even weirder to me than the -= does

I'm wondering if some other syntax might be cleaner (either an alternate function like unsetFlag does or an extra arg of deletion keys in an update)

[forest fractal]

Basically storing entire documents in flags because there is no free "GenericDocument" directory for modules to introduce their own 😅

[idle grove]

Yup, just pointing out the lowest level stuff.

[narrow sky]

i know, i know - and unsetFlag does work with nested properties

[idle grove]

I've seen too much stuff bypassing the setFlag method to update a document's flags.

[native minnow]

Does updating a property still work "normally"?

[idle grove]

I will agree that the full path of foundry.data.operators.ForcedDeletion.create() is quite long for something that (dev side) used to be so short to grab.

[idle grove]

Yes, no change, just an escape hatch for a full force replace is added, useful for whole object replacement.

[forest fractal]

Gonna have to prepend every file interacting with data with const { ForcedDeletion, ...otherOperators } = foundry.data.operators;

[formal basalt]

also calling a create for a delete action is making my brain go crazy

so i will settle for the new on those for sure

[idle grove]

IMHO, it's should be a singleton, at least for client side, no meaningful value after all, it's just a marker for deletion.

[narrow sky]

The == prefix already existed, this supercedes this

[idle grove]

It's going to get serialized on the wire though.

[tawdry talon]

(added in v13 iirc)

[idle grove]

Huh, never knew that prefix existed.

[formal basalt]

it is new

and very useful

[idle grove]

Yea, I've not done a deep dive on v13.

[formal basalt]

hmm, OP was edited, i don't see the new ForcedDeletion() syntax in the code snippets anymore

or did i dream it? oO

[wheat wraith]

It's supported, but create is recommended as a factory especially for ForcedReplacement since that's what imbues it with proxy behavior that allows the replacement to be interacted with as it's underlying value.

[formal basalt]

gonna be honest, i understood half of what you said, but i believe you :p

[wheat wraith]

it would be theoretically possible to have one singleton instance of ForcedDeletion globally that is always reused:

globalThis._fd = new foundry.data.operators.ForcedDeletion();

Like:

model.updateSource({deleteMe: _fd});

[idle grove]

JS's Proxy object is all sorts of black magic, if you want to read about it: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Proxy
It's what backs things like VueJS's state/reactivity system.

[lofty oracle]

is ForcedReplacement and ForcedDeletion namely specifically for the == and -= operations given in the example? I wasn't aware that such things existed, and I'm not sure how my module's code might be affected by this change.

[narrow sky]

Should add ForcedCreation.delete to complete the set

[topaz hornet]

If you weren't using -=key to delete a key or ==key to overwrite it in an update, you would be unimpacted

[lofty oracle]

got it

[tawdry talon]

Updates generally merge objects. If you want to replace the object wholesale you presently use ==
Updates simply add a key if needed and change the value. They don't remove keys normally. To do so you presently use -=

This is a replacement for the above so if you don't presently use either it doesn't affect you

[wheat wraith]

Checking Whether You Are Impacted

The following regex pattern is a good one to use to look for possible usages of a legacy forced deletion (-=) or forced replacement (==) key:

["'`\.][=-]=

[river fulcrum]

I personally really like working with Arrays (not so much in foundry) and I have a feeling I'll see a lot of use for them lol

[formal basalt]

those keys can be inside a string though, not necessarily at the start of one

[tawdry talon]

And don't forget about template literals 😛

[formal basalt]

especially with a setFlag, you would have it inside the string quite often

so you need to look for .-= and .== too

[narrow sky]

there will be deprecation warnings, right?

[wheat wraith]

yes.

[topaz hornet]

I think a regex search of

["'`\.][=-]=

should match all the expected occurrences (three types of quotes or periods followed by -= or==)

[river fulcrum]

I agree on the force deletion singleton. Would make it a little more intuitive to use since it's "just a marker". Having a factory function makes sense for the replacement class

[dull elm]

So if I use a template literal for a deletion, e.g. this.item.update({ [system.actions.-=${deleteId}]: null }); will it still be viable with new format?

[idle grove]

That's what is going away in v16.

[river fulcrum]

until v16.
you'll want this.item.update({ [system.actions.${deleteId}]: foundry.data.operators.ForcedDeletion.create()}); starting with v14

[odd token]

im still waking up, so this is more of a question than a suggestion.

Is this a potential use of a Symbol? It has some similarities to strings in that they can be compared easily, but would allow the creation of that proxy (i think) via stuff like Object(symbol)

[idle grove]

Template literal or not, it's still results in a string using the deletion key.

[river fulcrum]

or better yet

this.item.update({ 
  "system.actions": {
    [deleteId]: foundry.data.operators.ForcedDeletion.create()
  }
});

[shell escarp]

I am realizing one flaw of this. I can no longer impose these operations via HBS.

E.g. for including hidden input that deletes some data that is not needed. Needs to be transitioned into form submission handling (which is a bit annoying for cases where such wasn't needed before).

Tho I'm not sure that even worked anymore because of the need for null value...

[river fulcrum]

a sort of related question (I don't actually expect any answers) but with the seeming addition of the concept of data operators, are there any plans to allow a more targeted manipulation of lists/arrays?

more targeted than manipulating the array locally and then forcing a replacement with the new array

[formal basalt]

true, you will have to work the form data manually in the app

[wheat wraith]

Discussed with the team, we will add the following globalThis properties to provide a more slim syntax for invoking these operators:

  /**
   * A singleton ForcedDeletion operator instance that can be reused.
   * @type {foundry.data.operators.ForcedDeletion}
   */
  _del: new foundry.data.operators.ForcedDeletion(),

  /**
   * A reference for ForcedReplacement.create that can be easily referenced.
   * @type {foundry.data.operators.ForcedReplacement.create}
   */
  _replace: foundry.data.operators.ForcedReplacement.create,

[idle grove]

Pinned that and the original post, for easier pointing to later.

[river fulcrum]

y'all are awesome

[topaz hornet]

I think the globalThis references will help a lot. With that, it's similarly idiomatic to the existing syntax (which isn't bad once you learn it to begin with).

Especially if it gets documented in the KB so new devs have some way to actually learn it in the first place

[distant badge]

Overall: I like this change, I think it's much nicer than the key-based implementation.

Specifically: the example provided of "resetting back to initial value" with:

model.updateSource({someField: foundry.data.operators.ForcedDeletion.create()});

feels really odd to me because of the factory being named .create because it's kinda weird reading "ForcedDelete, but create it"; however using the _del blessed global feels better to me, so it's a really minor thing IMO, as I'd likely end up using _del and _replace in every situation where I use them, which isn't super frequent at least for me.

[maiden fox]

I know create is used throughout the code, but would a name-change like init make more sense?

implies that the ForcedDeletion is being initialized, and not that anything is being created

[silk sundial]

Feels like with _del and _replace being added it's not worth breaking convention

[topaz hornet]

I suspect staff will want to stick to their existing convention and encourage people to use the global references for an alternate phrasing

[maiden fox]

yeah makes sense to me

[distant badge]

Yeah, I definitely think sticking to convention is a good reason, it was just something my brain got caught on initially, without the blessed global I do think it could be worth considering alternatives, but the blessed global is a good alternative IMO

[formal basalt]

init doesn't have the same purpose, you init an instance, create is to create said instance

[maiden fox]

tautological explanation but i think i get what you mean

[mint spoke]

Post-operation workflows like _onUpdate or update hooks see DataFieldOperator instances in the provided diff.
I'm assuming an adjusted update hook would look like this:

Hooks.on("updateToken", (doc, changes) => {
  const myObj = foundry.utils.getProperty(changes, "flags.myModule.myObj");
  if (!myObj) return;

  // Old syntax.
  // if (myObj["-=myKey"] === null)

  // Incorrect: Global _del is not required to equal custom deletion instances.
  // if (myObj.myKey === _del)

  if (myObj.myKey instanceof foundry.data.operators.ForcedDeletion)
  { /* React to deletion */ }
});

If not, an example of this use case would be nice.

[idle grove]

Strict equals won't work the second someone calls new ForcedDeletion in some module/system.

[formal basalt]

instanceof ForcedDeletion

[idle grove]

Someone will do it, because what can go wrong, will go wrong.

[mint spoke]

Ok, adjusted the example for anyone finding it as a search result

[wheat wraith]

If you need to identify and respond to forced deletion specifically, an instanceof check is the correct way

This is usually unnecessary though, most of the time all you need to know is “this thing changed”, for which a (“myKey” in object) test is preferred

[scarlet marlin]

Writing out _delete for the global might be nice to match with _replace, rather than having one with the full word and the other truncated

[formal basalt]

maybe the idea is one is a singleton/variable, the other is a function to call

a different syntax to differentiate those

[wheat wraith]

Yeah, was thinking this. Might change it

[solid night]

Why are the names prefixed with Forced?

[shell escarp]

They're not discarded for any reason. E.g. via diffing.

[shell escarp]

Does this BTW finally mean that {nullable:false, required:false} numberfield gets cast to delete operator if it's null? In formdataextended handling.

Since supporting such in sheets is kinda awkward, since formdataextended gives null for empty field.

[wheat wraith]

Hmm that doesn’t happen automatically, but I see what you’re saying.

I’m not sure whether a form should ever parse directly to a special data operator, I think that is probably too strong an assumption.

[shell escarp]

It would then have to be in the documentsheet somehow.

[wheat wraith]

We might be able to have some sort of database operation preprocessing though where explicit undefined could get cast to a ForcedDeletion operator maybe

[shell escarp]

That undefined to delete handling would be seriously breaking change since undefined previously was guaranteed to not do anything. And I think datamodel cleandata gives undefineds for such, which could cascade to something nasty.