#Data Synchronization across servers with Redis

1 messages · Page 1 of 1 (latest)

orchid phoenix
supple parrot
#

yeah btw aglerr im thinking of a solution

orchid phoenix
#

I changed the repo name and make it public

#

Okay, let's hear it.

supple parrot
#

hmm

orchid phoenix
#

Can I keep cache on the redis?

#

Like load the clan on server startup, and if the clan is not on redis cache, load it from the MySQL.

wary mango
#

Database -> redis
you shouldnt really use redis as database...

orchid phoenix
#

I couldn't think of any different solutions right now

supple parrot
#

Usually,
get data:
if data is not in local cache -> check redis cache -> if not in redis cache -> check persistent database -> if in p-db but not in redis-cache put in redis-cache

#

tho it requires you to be disciplined

#

the hard problem is to avoid invalid data

orchid phoenix
#

Should the data inside the redis cache expires?

#

There's Jedis#expire apparently.

supple parrot
#

thats reasonable

#

I usually had it to be 10 min or smtng

orchid phoenix
#

And somehow we need to save the data to mysql too before it expires.

supple parrot
#

I believe you can attach an expiration-listener

#

if not

#

whenever you update the cache, update the mysql database also

orchid phoenix
#

The bedwars clan data will be changed very often I imagine, isn't that will be expensive?

supple parrot
#

I mean do it concurrently

#

okay aglerr

#

I think you could do it like that and

#

in addition, whenever a foreign server sends an update, grab the updated data directly from the rediscache

orchid phoenix
supple parrot
#

yes but also a local one

orchid phoenix
#

And use this system right?

if data is not in local cache -> check redis cache -> if not in redis cache -> check persistent database -> if in p-db but not in redis-cache put in redis-cache

#

And cache it on the local when server startup?

supple parrot
#

I think so

orchid phoenix
# supple parrot I think so

What about try to load clans on PlayerJoin, so when player join get the clan uuid and if the clan is not on local cache/redis cache, load it directly from mysql

supple parrot
#

First and foremost, use APPLE event but yeah

orchid phoenix
#

I don't think I want to loop through all clans, and also it has multiple tables which is kinda confusing.

#

What is APPLE event?

supple parrot
#

AsyncPlayerPreLoginEvent

orchid phoenix
#

Oh okay, lmao.

supple parrot
#

let me write some cases for you

#

actually

#

what data are you saving?

#

or well

#

managing?

#

just clan data?

#

or player data also?

orchid phoenix
#

player data for the rank in the clan, and what clan they are in

supple parrot
#

link the classes whose job is to carry the data

#

like from the gh

orchid phoenix
#

that's the tables

supple parrot
#

Don't you have a data-transfer-object?

orchid phoenix
supple parrot
#

right

orchid phoenix
#

also, i use player name to identify the player, don't be mad tho 😅

supple parrot
#

👀

orchid phoenix
#

all of the uuid is clan's uuid

supple parrot
#

Okay

#

When a user joins a server, load the data

┕-if not in redis cache
 ┕-queue persistent database
   ┕-put result in redis cache
 ┕-put result in local hashmap
┕-get result from local hashmap```
orchid phoenix
#

I need to load the clan's data too right?

supple parrot
#

yeah not done xD

orchid phoenix
#

I mean alongside loading the user's data.

#

(If the user has a clan)

supple parrot
#

havent written about clan data

orchid phoenix
#

Is it like that?

supple parrot
#

Gonna write some concerning clans

orchid phoenix
#

Also I make it method return boolean, so we know If the loading fails or not

supple parrot
#

No no no

#

Throw an exception instead

orchid phoenix
#

I already do that

supple parrot
#

Yeah no need for that bool

orchid phoenix
# 
    @EventHandler
    public void onAsyncPreLogin(AsyncPlayerPreLoginEvent event){
        if(!clanCache.loadPlayerSync(event.getName())){
            event.setLoginResult(AsyncPlayerPreLoginEvent.Result.KICK_OTHER);
            event.setKickMessage("Failed to load your data");
        }
    }
``` I want to do something like this, or just make the method void?
supple parrot
#

That’s probably fine

orchid phoenix
#

okay great!

supple parrot
#

Anyways

#

I’m just doubtful whether it’s smart to keep all clans loaded in the local hashmap as well in the redis cache

orchid phoenix
#

I don't know what are the limitations of the Redis Cache.

orchid phoenix
#

@supple parrot If you save user's data when they leaves, do we need to use pub/sub for that? I mean you have saved it on redis cache when player leaves.

supple parrot
#

Yes it’s smart to save it there

#

and use pub sub

#

Such that other servers can pickup any changes

orchid phoenix
#

Do I need to send a message like aglerr;clanuuid;rank and then split it or something.

orchid phoenix
#

@supple parrot

supple parrot
#

send code

wary mango
orchid phoenix
orchid phoenix
#

Can you just GSON#toJson(clanPlayerObject) and vice-versa?

#

Should I create a single and keep the reference of Gson object, or just use new Gson everytime I needed it?

#

On publish

try(Jedis jedis = redisHandler.getJedisPool().getResource()){
  jedis.auth(redisHandler.getPassword());
  jedis.publish(RedisHandler.CHANNEL_CLAN_PLAYER, gson.toJson(clanPlayer));
}

On subscribe

ClanPlayer clanPlayer = gson.fromJson(message, ClanPlayer.class);
clanPlayerMap.put(message, clanPlayer);
wary mango
wary mango
orchid phoenix
#

I have a field for the gson

#

private final Gson gson = new Gson();

#

That's pretty useful ngl lmao

#

You can easily convert object to string in json format and convert json format to the object itself

orchid phoenix
#

Oh wait, I could just save to the Redis with Gson

#

Oh wow, this is perfect

orchid phoenix
#

I'm so happy ajdnjawndawndsadwk

orchid phoenix
#

But my friend says, it's better to store the data one by one to Redis instead store it as a Json @wary mango, what do you think?

supple parrot
#

Who’s your friend?

#

Anyways that might be smart

#

Altho should be said redis is nosql

wary mango
orchid phoenix
orchid phoenix
wary mango
#

Ah ok

wary mango
orchid phoenix
# wary mango Wdym by " one by one"? 
jedis.hset(key, "name", clan.getName());
jedis.hset(key, "tag", clan.getTag());
.... and others

vs

jedis.hset(key, "details", gson.toJson(clanObject));
wary mango
#

aha

#

But isnt message a single string?

orchid phoenix
#

Yes, I think I use both on pub/sub and in general.

#

And my friend said that If I use String Json to store it, it would make the Redis heavier.

#

Which I think it doesn't make sense.

#

Because we're storing a String, unless the length of the String matters.

supple parrot
#

It doesn’t

#

Redis is mainly focused to be a high performant key-value cache

supple parrot
#

Of course storing stuff in memory requires memory

#

But the alternative is to store more entries

#

and also you don’t need json, believe jedis or well redis in general has support for maps,sets and so on

orchid phoenix
#

It's really doesn't matter right? whether I use Gson or not, it's the same.

supple parrot
#

Yeah both work finely

orchid phoenix
#

Okay

#

@supple parrot now I'm struggling to when should I load the clans data, Clan, ClanStats and ClanBedwars.

#

Should I load it at the same time as we load player data?

# 
APPLE event after player data is loaded
  check if player has a clan
    if player has a clan, load the clan from redis cache
      if clan data on redis cache not exist
        load from mysql
        store it on redis cache
        store it on local cache
      if clan data on redis cache exist
        load from redis cache
        store it on local cache
orchid phoenix
#

Does it sound safe?

supple parrot
#

Hmm yeah

#

And when you update the clan data

#

Send and update message to other nodes

#

and let the other nodes re-fetch the data from REDIS/MYSQL

orchid phoenix
#

Gotcha. I have updated the code, feel free to give me any advice, and I'm going to sleep now. Thanks y'all.

supple parrot
#

Sounds delightful, sleep tight

orchid phoenix
#

I'm back and fresh, and now It's the time to create a method to create a clan, invite/kick/promote members, and changing clan data such as tag, colorTag, etc with command.

#

Right now I don't know how to generate uuid for the clan and making sure the generated uuid is not taken by other clan yet. I know I should use UUID#randomUUID but how do I get all existing clan uuid.

wary mango
#

The chance that uuid will be the same is soo minor

orchid phoenix
#

Oh okay

orchid phoenix
#

@supple parrot how can I keep detect If the members of the clan is online or not? Should I use Redis for that too? And also for ClanChat, should I use plugin message channel or redis.

orchid phoenix
#

umm he left 😢

#

@wary mango any chance you know this?

wary mango
#

it is much better at messaging than bungeecord one

#

For online players, i'd make a plugin for bungee

orchid phoenix
wary mango
orchid phoenix
wary mango
#

break down your project into bukkit and bungee module and abstrsact out platform specific stuff. Like, clans are going to exist on all servers and proxy aswell. Then, to get all online players, you send a message to bungee snd wait for response

orchid phoenix
#

Oh so just check If it's only requested?

wary mango