Syncing two collections, on different installations of Appwrite

[plain breach]

I have a collection DATA, and a collection called SYNCS. Also a third collection that holds a FLAG.

Also, I have two functions, A function, will catch events on collection DATA, and write changes on SYNCS collection, to put them on a queue. if the FLAG is ON, it will ignore the events of DATA changes.

Function B executes every X time, it will first, put FLAG - ON, and then make the data changes that were saved until now on SYNCS. Then it will put FLAG - OFF.

Why to put FLAG - ON? Because, I want to ignore the events created by the changes made by function B.

The problem. The Function A (the one that is triggered by an event) always is executed after Function B finishes, so FLAG is OFF by that time.

So the events created during the function A, seems to be executed only after function B is finished.

Any suggestions on how to handle this, will be really appreciated.

[quaint sphinx]

Thanks for shifting it here @plain breach

Tagging @surreal ocean and @raw hemlock to keep them in loop

[plain breach]

Thank you @quaint sphinx. I didn't open an GitHub Issue because I did not knew how to tag it

[exotic vale]

Hey do you have any code sample for this?

[surreal ocean]

Is the problem that server A has data and it gets synced to B, but then B tries to sync that back to A creating a loop?

[plain breach]

Huge code right now, because I'm testing lot of things.

[exotic vale]

Fair enough

[plain breach]

Think like, server B (remote) gets new data, a function catch that data with an event, and put it on a collection called SYNCS, that collection is a queue to sync to server A (local).

On server A there is a function that set flag ON in server B (remote), so server B will ignore new events (during the execution of this function).

Then, the same function starts syncing data on queue (SYNCS collection) the problem, and finish the process setting the flag OFF.

Problem: The Function A (the one that is triggered by an event) is executed after Function B finishes, so FLAG is OFF by that time, so it will generate new SYNCS documents, that is not the intended.

I could try to make a diagram if you guys need to understand better

[surreal ocean]

let's take a step back and don't think about your implementation...what are you trying to do?

[plain breach]

That's a good Idea.

I have two appwrite instances, one running locally, and another on internet (call it remote).

In both, there is a collection, and I want to sync bidirectionally this collections. Like, if I add something to the collection on the remote server, it will replicate the same on local. The same happens in local, if I add something on its collection, it will replicate that on remote.

This have to work for CREATE, UPDATE, DELETE

[surreal ocean]

And so if a document is synced from remote to local, you don't want that change to get synced again from local back to server in an infinite loop, right?

[plain breach]

Yes sir!

[surreal ocean]

Right so you need a way to know there's no need to update...

I think you're going to have to store something on the document to know. Maybe you can store who updated the document: sensor or sync. If it was updated by sync, don't sync

[plain breach]

I though on the same, so I will implement that, to check how it works with that

but actually, reading from you that work around points me in the right direction

thank you so much Steven

And sorry for my late reply, working a lot today!

[surreal ocean]

no worries!

[lusty yew]

@plain breach everything good? Checking to see if it makes sense to close this and mark as solved 🙂

[plain breach]

It's needed to mark it as solved?

[lusty yew]

Checking if everything is good and if you still needed any assistance

[plain breach]

Just as a little update on this.

IDK why this happens, if is just me, or what. buuut.

Seems like the time passed from the request made with databases.update_document('[DATABASE_ID]', '[COLLECTION_ID]', '[DOCUMENT_ID]') and the execution of a function that catch that event is around 6.5 - 7.5 seconds.

What I mean with this.

If I set the a flag ON, to stop catching events, this flag should be ON for at least that amount of time, before going OFF, otherwise, it will be useless.

[surreal ocean]

events get queued and, on self hosted, they're executed serially. So, if you have other events going on, they may be processed first before this update document one is processed

[plain breach]

any way to do it faster? or thats just how it works

[surreal ocean]

You would have to scale out horizontally with more workers that can handle the events

[plain breach]

Any guide on how can I do that?

I really want to try that out Steven