The type of actions you’re storing | SpigotMC | Page 1

granite panther Jul 3, 2022, 9:34 PM

#

I'm essentially just making a generic variable counter plugin our server uses - it's the simplest solution to our problem of storing custom-statistics about players without needing to come up with a storage-strategy for each statistic.
E.g.:

EASY_POT_BREWS (custom potions plugin, doesn't use default brewing stands or call default potion brewing events)
EASY_POT_BREW_NAME
KILL_CUSTOM_MOB
ATTEND_CLASS_ABOUT_POT_NAME
DISMOUNT_BROOM

etc etc.

Essentially each player has a map of string=int with different string keys we want to track information about.

Currently we store it in a fork of a plugin called HuskSync (which we bought & with the authors permission). HuskSync syncs player data (inventory, health, advancements - only in-game stored things) across multiple BungeeCord servers.

I've forked the plugin to also store our custom variable_counter data (a map of string=int for each player).

We store it terribly in a column called variable_counter along side all their other HuskSync stored data: https://github.com/Potterverse/HuskSync/blob/master/common/src/main/java/me/william278/husksync/proxy/data/sql/MySQL.java#L45

And we "serialize" this map into a string poorly too: https://github.com/Potterverse/HuskSync/blob/master/api/src/main/java/me/william278/husksync/bukkit/data/DataSerializer.java#L28

#

.

#

Wow I just wrote a whole ass essay in response and Discord just deleted it and made the thread

rugged rover Jul 3, 2022, 9:35 PM

#

I would suggest you cache the stats locally and save them on disconnect/server swap

#

Also

#

I hate that storage method

#

Lmao

#

It’s awful

granite panther Jul 3, 2022, 9:38 PM

#

which one

rugged rover Jul 3, 2022, 9:38 PM

#

Where you serialize it all into one glob

granite panther Jul 3, 2022, 9:38 PM

#

oh wait do you see my msg????

#

i can't see it i thought discord deleted it i was just retyping it lol

rugged rover Jul 3, 2022, 9:38 PM

#

Yeah it’s there

granite panther Jul 3, 2022, 9:38 PM

#

tl;dr:

We want to store a map of String, Integer for each player.
The String is a custom "stat" we want to track related to our server, e.g. potions_brewed or whatever. Sometimes there are more ideal solutions to tracking these stats and we could group them as some of the keys are related - but we don't have the time.
We currently do this through a fork of HuskSync. HuskSync is a premium resource (we paid for) and we have forked to store our String, Integer for each player.
This way the plugin takes care of loading the stats on join and saving them on quit for us - it uses redis messaging, etc.
Our fork stores in HuskSync by converting the String, Integer map into a string of key=value,key=value and then storing this in an extra column with other HuskSync data: https://github.com/Potterverse/HuskSync/blob/master/common/src/main/java/me/william278/husksync/proxy/data/sql/MySQL.java#L45

This strategy isn't sustainable as we don't have the time to continually rebase on HuskSync's changes & so I'm looking at how we can make it independent.

Obviously the ideal solution is caching and loading like they do ^ (their Redis messaging system makes up for the fact that onJoin is called on the new server before onQuit on the current server - which makes loading on Join and saving on disconnect a bit more difficult in practice), but we don't have tons of time to develop/maintain the system.

So essentially just looking to take the storing of our String, Integer map about each player into a different plugin

rugged rover Jul 3, 2022, 9:38 PM

#

You should make a separate table for this

granite panther Jul 3, 2022, 9:38 PM

#

well that's what I just retyped

#

yes ideally if we had the time to do it we'd even make related data into their own table. e.g. easy_potion_id, easy potion, year - and then for individual players we can just say easy_potion_id, number_of_times_brewed

but we don't have time so we took the low maintenance approach of instead storing easy_potion_name, number_of_times_brewed and potion_year, number_of_times_brewed

#

i know what the ideal solution is - but i also know we don't have time to develop it - so this is the next best (subsequently, worse) thing

rugged rover Jul 3, 2022, 9:41 PM

#

I would argue you’re going to spend more time in the long run trying to make this Frankenstein storage system work than you would doing it properly in the first place

granite panther Jul 3, 2022, 9:41 PM

#

the problem with the current system is that sometimes stats reset for a reason we haven't been able to track down (using the husksync fork) - but data like inventory/etc is unaffected (as it's also saved in game and not just in maps and the table)

rugged rover Jul 3, 2022, 9:41 PM

#

Can I tell you why you’re experiencing data loss?

granite panther Jul 3, 2022, 9:41 PM

#

if you can that'd be amazing

#

but we've probably ruled it out

rugged rover Jul 3, 2022, 9:41 PM

#

It’s a race condition accentuated by the fact that you’re storing it all in one column

granite panther Jul 3, 2022, 9:42 PM

#

why would that lead to the entire row in the husksync DB being deleted?

rugged rover Jul 3, 2022, 9:42 PM

#

The whole row is gone?

granite panther Jul 3, 2022, 9:42 PM

#

yeah

#

but the other data - e.g. inventory/stats isn't affected, because the server also stores them

#

so it's just our variablecounter data which is stored in the row and the map in the plug which is lost

#

even tho the entire row is deleted

rugged rover Jul 3, 2022, 9:43 PM

#

Well of course inventory isn’t, totally separate system

granite panther Jul 3, 2022, 9:43 PM

#

ye but we store the inventory in the same table on the same row

rugged rover Jul 3, 2022, 9:43 PM

#

Are you sure the row existed to begin with?

granite panther Jul 3, 2022, 9:43 PM

#

ye

rugged rover Jul 3, 2022, 9:43 PM

#

Wait what

#

If the inventory isn’t lost then the row isn’t lost

#

If the inventory is in the row

granite panther Jul 3, 2022, 9:43 PM

#

... the row is lost

#

like trust

rugged rover Jul 3, 2022, 9:44 PM

#

granite panther ye but we store the inventory in the same table on the same row

Explain

granite panther Jul 3, 2022, 9:44 PM

#

https://github.com/Potterverse/HuskSync/blob/master/common/src/main/java/me/william278/husksync/proxy/data/sql/MySQL.java#L23 this is the table

#

when player swaps server, all that data is saved from prev and loaded onto next

rugged rover Jul 3, 2022, 9:44 PM

#

Right

granite panther Jul 3, 2022, 9:44 PM

#

and redis messaging handles the whole onjoin being called before onquit

rugged rover Jul 3, 2022, 9:44 PM

#

Sure

granite panther Jul 3, 2022, 9:45 PM

#

sometimes - with no particular reason that we've been able to find - the row is deleted

#

but the inventory is correct

#

location unaffected

#

health etc is correct

#

but variable_counter is deleted

#

when i say is correct

rugged rover Jul 3, 2022, 9:45 PM

#

So the row isn’t deleted, the variable_counter is

granite panther Jul 3, 2022, 9:45 PM

#

i mean it loaded before the row deleted

#

no if you look at the table

#

the row is gone

rugged rover Jul 3, 2022, 9:45 PM

#

I see

#

Do you ever delete rows?

granite panther Jul 3, 2022, 9:46 PM

#

not that i'm aware of

#

i appreciate that you're trying to help and that you're not able to validate anything of what i'm saying btw, i promise that what i'm saying is accurate to the best of my ability and i like to think im fairly competent

rugged rover Jul 3, 2022, 9:46 PM

#

How do you interface with the table

granite panther Jul 3, 2022, 9:47 PM

#

https://github.com/Potterverse/HuskSync/blob/master/common/src/main/java/me/william278/husksync/proxy/data/DataManager.java

#

https://github.com/Potterverse/HuskSync/raw/master/images/flow-chart.png

#

that flow chart probs helps more

rugged rover Jul 3, 2022, 9:48 PM

#

Nah I want the actual code

#

That’s the only place it happens?

granite panther Jul 3, 2022, 9:48 PM

#

oh it's all on that github

#

yeah im pretty sure everything that wants to interact with DB goes through that class

#

also yes i'm aware we're 43 commits behind HuskSync master - but that's part of the reason I'm trying to abstract this string=integer data storage into it's own plugin

#

just no time to maintain

rugged rover Jul 3, 2022, 9:49 PM

#

How confident are you that the row exists prior to them logging in and it isn’t just them having saved minecraft player data

#

Since you said the stats are empty

granite panther Jul 3, 2022, 9:50 PM

#

very because if that was the case the inventory wouldn't sync between the two servers but it does

#

when i say stats i mean variable_counter

#

by being a fork of husksync we get the added benefit of loading into redis and saving onto redis and handles all the issues of onjoin and onquit order

but variablecounter data keeps resetting for players (it's related to restarts/crashes - so it's not entirely random)

#

our current solution is kick players from the husksync servers 45 seconds before we restart

#

and that's stopped the variable counter data being lost

#

but when it crashes, the variable counter data for online players is lost

#

it doesn't crash often - once every few months - but still

rugged rover Jul 3, 2022, 9:51 PM

#

Question

#

When their row disappears

#

If they relog, what happens

granite panther Jul 3, 2022, 9:51 PM

#

new row is generated

rugged rover Jul 3, 2022, 9:51 PM

#

Can you consistently reproduce this

granite panther Jul 3, 2022, 9:52 PM

#

i haven't been able to get a step-by-step do this and this will cause it

#

if i had then i probably would've been able to fix it by now

rugged rover Jul 3, 2022, 9:52 PM

#

But can you reproduce it

granite panther Jul 3, 2022, 9:52 PM

#

yea

#

it's related to restarts/crashes

#

whenever one of them happens variable_counter data could be lost

#

for online players*

rugged rover Jul 3, 2022, 9:52 PM

#

Well that seems like you just aren’t saving it and it’s being cleared on login or something

granite panther Jul 3, 2022, 9:53 PM

#

I'm thinking that too but I swear I cleared that being a potential problem months ago

#

I'm double checking now to make sure my memory isn't failing me

rugged rover Jul 3, 2022, 9:56 PM

#

And this is on mysql?

granite panther Jul 3, 2022, 9:57 PM

#

rugged rover Well that seems like you just aren’t saving it and it’s being cleared on login o...

i can't see this happening

rugged rover Jul 3, 2022, 9:58 PM

#

@granite panther what happens if their row vanishes and then they swap servers

granite panther Jul 3, 2022, 9:58 PM

#

no idea

rugged rover Jul 3, 2022, 10:01 PM

#

I think I may know what your issue is

#

The only data being lost is your new custom addition right?

#

The rest is all preserved?

granite panther Jul 3, 2022, 10:02 PM

#

the entire row is being deleted

#

but yes

rugged rover Jul 3, 2022, 10:02 PM

#

I suspect the Husk code is doing it intentionally and has a mechanism for caching it’s own stock data in Redis that you haven’t updated to also preserve your custom data

granite panther Jul 3, 2022, 10:03 PM

#

why would that only happen in the event of players online in restart/crash

rugged rover Jul 3, 2022, 10:03 PM

#

I’m not familiar enough with the husk plugin to know exactly where that is, but that’s my suspicion as to what is happening

rugged rover Jul 3, 2022, 10:03 PM

#

granite panther why would that only happen in the event of players online in restart/crash

Because your server that has the state is going down before it has an opportunity to save it

#

That’s the only explanation for what’s happening

granite panther Jul 3, 2022, 10:08 PM

#

rugged rover I suspect the Husk code is doing it intentionally and has a mechanism for cachin...

Redis msg sent with player's uuid: https://github.com/Potterverse/HuskSync/blob/master/bungeecord/src/main/java/me/william278/husksync/bungeecord/listener/BungeeEventListener.java#L40
Redis msg sent with PlayerData object: https://github.com/Potterverse/HuskSync/blob/master/bungeecord/src/main/java/me/william278/husksync/bungeecord/listener/BungeeRedisListener.java#L72
PlayerData object updated to contain my custom var:
https://github.com/Potterverse/HuskSync/blob/master/common/src/main/java/me/william278/husksync/PlayerData.java#L49

redis classes:
https://github.com/Potterverse/HuskSync/blob/master/common/src/main/java/me/william278/husksync/redis/RedisMessage.java
https://github.com/Potterverse/HuskSync/blob/master/common/src/main/java/me/william278/husksync/redis/RedisListener.java

doesn't contain word "redis":
https://github.com/Potterverse/HuskSync/blob/master/common/src/main/java/me/william278/husksync/proxy/data/DataManager.java

#

as much as i would love for the explanation to be that simple, it doesn't appear to be 😦

#

there are 100,000 possibilities here with the amount of states being managed and message being sent

#

this is why i just want to make the storing of String=Integer map independent, but then I either have to just suck up the potential of 100 update/select SQL statements being called a second (unlikely, but possible) or implement my own system of caching to fix handling onjoin vs onquit order

#The type of actions you’re storing