Error while trying to use Loro CRDTs (Uncaught ReferenceError: FinalizationRegistry is not defined)

[nocturne pumice]

Hey guys im trying some experiments with Loro and Convext together (https://www.loro.dev/docs/tutorial/get_started) but am running into an issue.

Here is a basic query I am trying to write:

export const myQuery = query({
  args: {},
  handler: async (ctx, args) => {
    const doc = new Loro();
    doc.getList("mylist").insert(0, "A");
    return doc.exportSnapshot();
  },
});

The error I get is:

✖ Error: Unable to push deployment config to https://kindly-elephant-254.convex.cloud
400 Bad Request: InvalidModules: Hit an error while pushing:
Loading the pushed modules encountered the following
    error:
Failed to analyze auth.js: Uncaught ReferenceError: FinalizationRegistry is not defined
    at <anonymous> (../node_modules/loro-wasm/bundler/loro_wasm_bg.js:209:9)

I suspect the issue is because loro uses wasm. I have read however that WASM is supported in both runtimes.

Is this not correct?

[lost oar]

Wasm should work in general. The error seems to be about FinalizationRegistry https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/FinalizationRegistry
which Convex runtime does not support because it is nondeterminstic (it relies on knowing when garbage collection runs) so it messes with query subscriptions and mutation conflict detection which rely on determinism. I don't know why Loro uses FinalizationRegistry (it shouldn't be required to use wasm) so it may be possible to avoid the usage or hack around it if not.

[nocturne pumice]

ah okay, thats a shame but it makes sense, thanks Lee

[lost oar]

It's also possible we can work around this on our side. I started an internal discussion. Thanks for reporting!

[nocturne pumice]

BTW for anyone that stumbles across this in the futurte, I raised this issue with Loro here: https://github.com/loro-dev/loro/issues/248

[stray field]

Hey @nocturne pumice, I was interested in convex + local first but didn't see anyone using them by now, since Loro is kind of for local first apps I was going to ask if you had a chance to combine this stacks?

[nocturne pumice]

thats exactly what I was investigating until I ran into this issue

I would like to combine convex and local first, this seemed like a way of doing it

[stray field]

would love to hear about your experience if you find a way!

[red briar]

@nocturne pumice I'm curious if you thought about giving https://replicache.dev/ a try?

[stray field]

For some reason everyone is suggesting replicache but there is no single working example or implementation of replicache + convex
@red briar why do you think that replicache can work w/ convex?

[red briar]

I think conceptually it should work. But I have not seen anyone try it. Thus I was curious if @nocturne pumice had given it a try.

[nocturne pumice]

no I havent tried it, I think the fact that they charge for something I think I could implement myself put me off. Ill give it another look 🙂

[tulip nymph]

@icy linden - @lavish ridge made a demo recently: https://github.com/sujayakar/replicache-convex
He hasn't put it on convex.dev/templates but is working on it - let us know what you think

[stray field]

@lavish ridge do we still get type safety on the front end?

[nocturne pumice]

nice project @lavish ridge !

the amount of extra work you have to do is kind of annoying:

  replicacheClient: defineTable({
    clientId: v.string(),
    clientGroupId: v.string(),
    lastMutationId: v.number(),
    version: v.number(),
  })
    .index("by_client_id", ["clientId"])
    .index("by_client_group_id", ["clientGroupId", "version"]),

  replicacheServer: defineTable({
    serverId: v.string(),
    version: v.number(),
  }).index("by_server_id", ["serverId"]),

But I guess this is the price you have to pay if you want to ensure conflict free and offline-first. I suspect I was going to have to do something simmilar with Loro. Its a shame that I cant work around that FinalizationRegistry issue

[lavish ridge]

@stray field, it needs some work to cleanup the types and API, but I think typesafety is mostly there. the replicache tutorial has me pass in the Message type directly to tx.scan on the client side, and then we use the MessageWithID type in the mutator that then would connect to the mutation type on the server.

and yeah @nocturne pumice, the hope would be that most of this could be abstracted away in a convex-replicache library (or... component 😃 )

related: I just submitted a PR to add no-op implementations of FinalizationRegistry and WeakRef to our isolate environment, so hopefully loro can work after that!

[nocturne pumice]

oooo! ❤️ that would be awesome!

[nocturne pumice]

ye sound like a good candidate for the upcoming convex modules (or whatever its going to be called) project

[lavish ridge]

i’ll update the thread when it’s in prod!

[stray field]

@nocturne pumice why you chose Loro over other lo-fi techs?

I quickly take a look at their website and saw that this tech is mostly suitable for text editing apps where LWW(Last Write Wins) and how does it work in the case of authoritative serves where some logic is included, for example for the systems like POS?

[nocturne pumice]

I dont know, I just wanted to tinker with Loro, I like the idea of CRDTs and it looked like a really nice and simple implementation. Im not convinced convex is neccessarily a good fit for it TBH.

Its probably a lot lower cost if you just run your own node server or use a cheaper FaaS platform like Cloudlfare workers or something.

The reasoning is that we dont really need the transactional gaurentees of convex as the CRDTs take care of that for us.

But im a Convex fanboy so everything looks like a nail when you have a Convex hammer 😉

[nocturne pumice]

ye its always going to be a tradeoff based on usecase

Its not always text applications that CRDTs work for tho. Think Figma, Canva, Google Sheets, etc etc

[stray field]

definitely, but all these apps you listed work w/ the same principle LWW right?

[nocturne pumice]

(well google sheets is OT based but you know what I mean)

[nocturne pumice]

im not sure about LWW to be honest, CRDTs just gaurentee that there wont be any conflicts when the two timelines are merged together

I was going to experiment around with it a bit more but my understanding is that user A can make changes to a list of items while user B makes changes to the same list. The list's final state will be a combination of those changes NOT just whoever made the last change

Loro site has a bunch of examples you can play with (https://www.loro.dev/docs/tutorial/time_travel)

[lavish ridge]

stub implementations of FinalizationRegistry and WeakRef are in production now! let me know if you run into any issues.

[nocturne pumice]

wooo! Ill take a look when I can ❤️

[nocturne pumice]

sadly I now get this error 😦

[lavish ridge]

huh very interesting. do you have a small repro @nocturne pumice? or is just trying to use loro sufficient?

[nocturne pumice]

yep simply trying to use loro in any mutation or query or action does it

I can give you an example but it would be like 1 line

new Loro()

[lavish ridge]

heh fair enough. will give it a shot 🙂

[nocturne pumice]

@lavish ridge did you ever get to the bottom of this?

[lavish ridge]

I did a bit of digging but am currently a bit stuck. I think the core issue is that we set up WASM imports like Cloudflare Workers (see the WASM snippet in https://developers.cloudflare.com/workers/wrangler/bundling/#files-which-will-not-be-bundled), but loro uses another convention where the ESM import is already instantiated and linked (see screenshot, taken from https://www.npmjs.com/package/loro-wasm?activeTab=code). cc @plush sphinx (we were just talking about this in person) and @oak locust

[nocturne pumice]

@lavish ridge Is there anything I can do to hack around it for now? Maybe fork and change something in Loro?

thanks for looking into this BTW I very much appreciate it!

This isnt mission critical for me BTW, just as a side-interest

[lavish ridge]

of course! @oak locust’s looking into some bundler changes we can make to get this working.

[left pasture]

@oak locust @lavish ridge any updates on using Loro ☝️ ?

[oak locust]

I've got a PR sitting that implements the other kind of wasm bundling, haven't looked at it for a while

[left pasture]

Does that mean its sorta a wont do? Should I scrap Loro, picking a conflict resolution system is business critical for us, need to make a call soonish. Would prefer to use Loro than Yjs - might have to even use automerge lol

[lavish ridge]

hey @left pasture, yeah this WASM stuff has been a big pain on our side (ecosystem hasn't stabilized on one import mechanism yet) -- from the last time we looked at it, I think it'd be a bit of work to get something stable working for y'all. I just tried it again, and it looks like their wasm loading also requires top-level await, which we also don't support yet.

@tulip nymph wrote up a doc on getting automerge working within convex (https://stack.convex.dev/automerge-and-convex#using-automerge-in-a-serverless-environment), and he also has a great post on a higher level text editing component that uses prosemirror (https://www.convex.dev/components/prosemirror-sync).

[left pasture]

What's everyone's general thoughts on automerge re performance and devx etc..?

[frigid sentinel]

I'd be very cautious about relying on CRDTs too much. it almost never works out in practice.

it can work very specific use cases really well, but if you try to solve too many application problems with them, you run into a lot of problems where mapping the data + change semantic domain to CRDTs is super awkward, and produces really large documents that essentially become journals of operations that have to be applied, are expensive to write to disk over and over again, and are super difficult to migrate

[left pasture]

Yeah tbh I ran into this problem pretty quick with Yjs, hence the switch to Loro, because you can shallow snapshot and cull unused history effectively. But yeah, what you're saying totally makes sense, in that case what's a solution for complex collaborative structures using convex? - I'm not talking a simple text document, but a collaboration use case with multiple collaborative datastructures.

[frigid sentinel]

where you want offline? like, optimistic behavior I take it?

b/c otherwise vanilla convex is okay

[left pasture]

Nah offline isn't that important tbh, just that users are able to make conflic free collabs

[frigid sentinel]

oh, vanilla convex will do that

[left pasture]

I don't see how acid transactions alone solve the problem of timing

[frigid sentinel]

they're strictly serializable, so they never race

but if you want optimistic update, this we haven't shipped will help a ton ( https://stack.convex.dev/object-sync-engine ) and in the mean time, replicache is still some of the best tech out there ( https://github.com/sujayakar/replicache-convex )

[left pasture]

this we haven't shipped
Meaning you will ship object sync at some point?

Yeah I guess you could say optimistic updates is the key, less the offline-first requirement. So just to confirm, if 2 users write to the same document, those writes happen sequentially, and its then last write wins?

[frigid sentinel]

No, not last write wins.

https://docs.convex.dev/database/advanced/occ