Data management pain | SpigotMC | Page 1

graceful slate Jul 22, 2022, 4:46 PM

#

@queen goblet Can i delete data from the cache on an asynchronous thread?

grave cobalt Jul 22, 2022, 4:46 PM

#

Yeah

#

Id advice using a ConcurrentHashMap

graceful slate Jul 22, 2022, 4:54 PM

#

thats what im using

#

thanks

graceful slate Jul 22, 2022, 4:54 PM

#

grave cobalt Id advice using a ConcurrentHashMap

is it ok if the objects inside aren't thread safe?

queen goblet Jul 22, 2022, 9:39 PM

#

No

#

They need to be too

graceful slate Jul 21, 2022, 10:20 PM

#

AutoFlush: Updates the database with the cache
https://mystb.in/EncouragedGenreQuoted.swift

Database: Class for interacting with database
https://mystb.in/GarlicOffenseThursday.php

DataListener: Listeners that loads and unloads players and families
https://mystb.in/AwesomeSubstituteKyle.swift

Family: A class that represents a family. A family is a group of players and has a leader. Should be loaded when at least one member of the family is online
https://mystb.in/ChainRailroadExcellent.java

PlayerData: A class that represents a player.
https://mystb.in/KissAddDaisy.java

Both Family and PlayerData have a static object called NULL_FAMILY and NULL_PLAYER_DATA. I use these because you can't use null values in a concurrent hash map. So basically these are used to tell Autoflush to delete the playerdata or family

#

I have two problems:

When something needs to access an unloaded player or family, the system caches the player/family until it is automatically unloaded on the next flush. I'd rather have some system that instantly unloads the player or family data after modification, but I have no clue how to implement such a system in a clean manner.
I need to preform two tasks that don't really make sense using this system. For one, I need to be able to pick a random family out of all existing families. I can't just pick a family from the database because it may be missing some families that are in the cache but not in the database. Second, I need to make a command that allows users to paginate through all families. No idea how to even start on that one

grave cobalt Jul 21, 2022, 10:40 PM

#

I'd rather have some system that instantly unloads the player or family data after modification, but I have no clue how to implement such a system in a clean manner.

#

I'd just stick with expiration or a garbage collection pattern

#

I need to preform two tasks that don't really make sense using this system. For one, I need to be able to pick a random family out of all existing families. I can't just pick a family from the database because it may be missing some families that are in the cache but the database. Second, I need to make a command that allows users to paginate through all families. No idea how to even start on that one

Are families separate from users?

#

but anyway

#

you must decide which data source is your primary data source

#

and then make sure that source is up to date when its needed

graceful slate Jul 21, 2022, 10:43 PM

#

grave cobalt I'd just stick with expiration or a garbage collection pattern

You think my current system is good?

graceful slate Jul 21, 2022, 10:43 PM

#

grave cobalt > 2. I need to preform two tasks that don't really make sense using this system....

Families are a group of users

#

they should be loaded in the cache if at least one member of the family is online

grave cobalt Jul 21, 2022, 10:44 PM

#

Idk

#

You think my current system is good?

#

hmm

#

how far are you planning to scale the system?

#

lets say amount of users your server should be able to handle concurrently

graceful slate Jul 21, 2022, 10:46 PM

#

maybe like 50-100

grave cobalt Jul 21, 2022, 10:49 PM

#

okay

graceful slate Jul 21, 2022, 10:51 PM

#

grave cobalt you must decide which data source is your primary data source

the cache is the primary data source i guess?

grave cobalt Jul 21, 2022, 10:52 PM

#

so what is the purpose of the database?

graceful slate Jul 21, 2022, 11:00 PM

#

its used to store data between server restarts obviosuly

#

I'm mainly loading the resources I will most commonly use from the database into the cache

#

and whatever is in the cache will eventaully get unloaded back into the database

graceful slate Jul 22, 2022, 2:15 AM

#

Ok forget the first question

#

I still need some help on the second

queen goblet Jul 22, 2022, 4:35 AM

#

graceful slate I still need some help on the second

whats the second one

graceful slate Jul 22, 2022, 4:36 AM

#

graceful slate I have two problems: 1. ~~When something needs to access an unloaded player or ...

here

queen goblet Jul 22, 2022, 4:37 AM

#

well

#

in that case if you need EVERY family

#

your likely better running a cache layer of your database and its expensive

#

its either that or make calls to get those users each time

graceful slate Jul 22, 2022, 4:37 AM

#

queen goblet your likely better running a cache layer of your database and its expensive

wdym

queen goblet Jul 22, 2022, 4:37 AM

#

queen goblet your likely better running a cache layer of your database and its expensive

IE on server startup you get EVERY family from database and cache it

graceful slate Jul 22, 2022, 4:38 AM

#

wont that be kinda expensive though

queen goblet Jul 22, 2022, 4:38 AM

#

yep

graceful slate Jul 22, 2022, 4:38 AM

#

do you think flushing the cache and then querying the database is also a good choice?

queen goblet Jul 22, 2022, 4:38 AM

#

thats the cost of needing to go through every iteration of your data

#

no

#

never flush your cache unless server shut down

#

your cache is a buffer between you and your database

graceful slate Jul 22, 2022, 4:39 AM

#

i think im using the term flush wrong lmao

#

i mean like

#

update the database with the cache without deleting it

queen goblet Jul 22, 2022, 4:41 AM

#

give me a minute

graceful slate Jul 22, 2022, 4:41 AM

#

this is my current implementation, not sure if it's good though

// In Family.java
@Nullable
public static Family loadRandomFamily() {
    CompletableFuture<Void> future = AutoSave.saveCache();
    future.join();

    Family family = LifeSpan.getInstance().getDatabase().getRandomFamily();

    if (family != null) {
        familyMap.put(family.getUuid(), family);
    }
    
    return family;
}

// In Database.java
@Nullable
public Family getRandomFamily() {
    try {
        PreparedStatement statement = conn.prepareStatement("SELECT * FROM Families ORDER BY RANDOM() LIMIT 1;");

        ResultSet result = statement.executeQuery();

        if (result.next()) {
            return formFamily(result);
        }
    }
    catch (SQLException e) {
        e.printStackTrace();
    }
    return null;
}

queen goblet Jul 22, 2022, 4:49 AM

#

If you start your server like this

#

#

then the line that the blue arrow points at should never happen

#

black = outbound red = inbound

#

read it from left to right top left corner

#

Hopefully this helps visualize seeing your cache as a type of buffer/layer between your program and database

#

ik my graphs suck lol

#

i dont have a good graphin program

graceful slate Jul 22, 2022, 4:56 AM

#

lemme read this

#

sorry im kinda lost on the second graph lol

#

"the event checks the.. if it already exists see red line"

#

what red line theres 4 connecting to it

#

@queen goblet

queen goblet Jul 22, 2022, 5:02 AM

#

#

there

#

arrows

#

maybe that helps?

#

@graceful slate

graceful slate Jul 22, 2022, 5:02 AM

#

oh damn thanks

queen goblet Jul 22, 2022, 5:04 AM

#

btw the 2 piece of text on the top center are one piece

#

"send it back to him and async update to database"

graceful slate Jul 22, 2022, 5:04 AM

#

ye i noticed that

queen goblet Jul 22, 2022, 5:04 AM

#

sorry it looks like a 5 year old made that graph 💀

#

but its the only way i can rly efficiently explain the buffer/layer cache system

graceful slate Jul 22, 2022, 5:05 AM

#

hmm so

#

if I'm understanding this correctly

#

you're saying that I should pretty much never read from the database aside from server startup?

#

and after the initial read I'm only ever updating the database?

queen goblet Jul 22, 2022, 5:06 AM

#

If your wanting a easy way to iterate through all your data, then yes

#

yep

#

you can async update task every like 5 mins

#

just run through all the cache asynchronusly and store it

graceful slate Jul 22, 2022, 5:06 AM

#

hmm

#

ok but heres the thing

#

I'm not iterating through everything very often

queen goblet Jul 22, 2022, 5:07 AM

#

what would you rather have

graceful slate Jul 22, 2022, 5:07 AM

#

so I guess its more of a choice then?

queen goblet Jul 22, 2022, 5:08 AM

#

Higher RAM usages (Maybe like 20MB at most? If your using pojo classes as the cache objects then cache is low asf) but instant gets, when someone opens a inventory to view familys everything is instantly there

Or use more CPU cycles every time someone opens a inventory, and they all wont instantly be there, they will load like a 1988 webpage

#

and not to mention have get calls to datbase players can control is dangerous

#

they can essentialsl ddos the mysql/entire server by spamming something that calls get to the database

graceful slate Jul 22, 2022, 5:09 AM

#

i see

queen goblet Jul 22, 2022, 5:09 AM

#

if its in ram its a mere instant return

#

you said 50-200 families right

graceful slate Jul 22, 2022, 5:09 AM

#

yeah

queen goblet Jul 22, 2022, 5:09 AM

#

yeah your cache size will probably barely be a few mb depending on the size of the objects your storing

#

if even a few mb

graceful slate Jul 22, 2022, 5:10 AM

#

and the entire database will be wiped every 3 days or so

queen goblet Jul 22, 2022, 5:10 AM

#

yeah you can still use cache layering

#

that wont matter to much

#

just if you wipe a database

#

wipe the cache too ofc

#

and stop any save tasks running

graceful slate Jul 22, 2022, 5:10 AM

#

ok so at this point I'm only using the database to keep data past server restarts

#

other than that, most of the work is in the cache?

queen goblet Jul 22, 2022, 5:10 AM

#

yep

#

yes

graceful slate Jul 22, 2022, 5:11 AM

#

alright

queen goblet Jul 22, 2022, 5:11 AM

#

A lot of discord bots actually cache all the servers theyre in

#

big bots like tickety have like 50gb of cache

graceful slate Jul 22, 2022, 5:11 AM

#

i thought this was a bad idea at first but now i dont think it'll be too expensive

graceful slate Jul 22, 2022, 5:11 AM

#

queen goblet A lot of discord bots actually cache all the servers theyre in

cough cough i do that too lol

queen goblet Jul 22, 2022, 5:12 AM

#

yeah you kinda have to with discord api sadly

graceful slate Jul 22, 2022, 5:12 AM

#

anyways

#

thanks so much i appreciate it 🙏

#

one more thing actually

#

about the paginating

#

how would I effectively paginate through the cache

queen goblet Jul 22, 2022, 5:13 AM

#

just a for loop is fine

#

and every x iterations make a new page

#

a page is a list of data objects

graceful slate Jul 22, 2022, 5:13 AM

#

but I'm not grabbing all the pages at once

queen goblet Jul 22, 2022, 5:13 AM

#

then about that idk

#

that ones more a logic question

graceful slate Jul 22, 2022, 5:13 AM

#

for example /family list 3 will grab the third page

#

yeah its fine i can figure that out

queen goblet Jul 22, 2022, 5:13 AM

#

"How do i organize a data set by pages"

Would likely be a decent search

Or when conclure/smile is on they might knoe

graceful slate Jul 22, 2022, 5:14 AM

#

I'm gonna have to sort the cache values by something and then find the page ig

queen goblet Jul 22, 2022, 5:14 AM

#

actually

#

if redempt is still in help-dev channel ask if his lib handles organizing HashMaps

#

might be able to rip a method from hit github

graceful slate Jul 22, 2022, 5:15 AM

#

hes not rn but maybe later

#

thanks so much

#

although I do want to keep this thread open and hopefully wait for 7smile7's answer

#

because he was the one who suggested me this original data management structure and i want to see what he says out of curiosity

queen goblet Jul 22, 2022, 5:16 AM

#

his is likely more or less of the same

#

he just explains it more programatically

graceful slate Jul 22, 2022, 5:16 AM

#

because how do huge applications do it then?

queen goblet Jul 22, 2022, 5:17 AM

#

clustering

graceful slate Jul 22, 2022, 5:17 AM

#

do huge applications store all the data in ram too?

graceful slate Jul 22, 2022, 5:17 AM

#

queen goblet clustering

what is that

queen goblet Jul 22, 2022, 5:17 AM

#

it really depends honestly

#

in minecraft you cant rly do it because well

#

its a game

#

but a huge banking system?

#

Yeah just get from datbase

graceful slate Jul 22, 2022, 5:17 AM

#

yeah speed doesnt matter there i guess

queen goblet Jul 22, 2022, 5:17 AM

#

who cares about the user waiting 2 seconds

#

but in a game, 2 seconds is a big difference hence why cache is a big deal

graceful slate Jul 22, 2022, 5:18 AM

#

I see

queen goblet Jul 22, 2022, 5:18 AM

#

thats why so many games have such high ram usages

graceful slate Jul 22, 2022, 5:18 AM

#

makes sense

queen goblet Jul 22, 2022, 5:18 AM

#

cause theyre caching files and assets

#

you would do this same structure if your working with files btw

graceful slate Jul 22, 2022, 5:18 AM

#

alright

#

I honestly might make a library that automatically does this stuff in the future

queen goblet Jul 22, 2022, 5:19 AM

#

I will say though, your only issue is gonna be data integrity, so those save tasks are 100% necessary and make sure the itnervals prolly no less than 5 mins

#

that way if something happens to your data only 5 mins of progress is lost

#

eg server crash

graceful slate Jul 22, 2022, 5:19 AM

#

yeah thats not too important for my situation

queen goblet Jul 22, 2022, 5:19 AM

#

solar ray makes your memory corrupt

#

not rly an issue now a days

#

but, bit flips happen

queen goblet Jul 22, 2022, 5:20 AM

#

graceful slate yeah thats not too important for my situation

it could be, so just make sure you have a save task always running

graceful slate Jul 22, 2022, 5:20 AM

#

do bigger games store data to the database after every modification though?

queen goblet Jul 22, 2022, 5:20 AM

#

i mean you could if you wanted to

graceful slate Jul 22, 2022, 5:20 AM

#

for example I play destiny 2 a lot and if I got a super rare drop but then my power went out a minute later and that super rare drop was gone I'd be pissed

queen goblet Jul 22, 2022, 5:20 AM

#

just dont do it on main thread

graceful slate Jul 22, 2022, 5:20 AM

#

surely the companies have a way to prevent that?

queen goblet Jul 22, 2022, 5:21 AM

#

graceful slate for example I play destiny 2 a lot and if I got a super rare drop but then my po...

well your cache is on theyre servers

graceful slate Jul 22, 2022, 5:21 AM

#

oh true

queen goblet Jul 22, 2022, 5:21 AM

#

so if your power goes out it wont matter

graceful slate Jul 22, 2022, 5:21 AM

#

but if their servers went out it would cause pain

#

got it

queen goblet Jul 22, 2022, 5:22 AM

#

yeah, you can definatly set a update() to your database after you update something in cache

Just as MC make sure its async or your gonna freeze the whole server haah

graceful slate Jul 22, 2022, 5:22 AM

#

okay

#

thanks so much man 🙏

#

i learned a lot from this

queen goblet Jul 22, 2022, 5:23 AM

#

ofc

#

also make sure your calls to your cache are on main thread, they likely already are but thought ide mention

#

i would definatly look into Completable Futures or callback functions tho

graceful slate Jul 22, 2022, 5:23 AM

#

i mean shouldn't it work on an alternate thread anyways

#

I'm using Concurrent hash map

queen goblet Jul 22, 2022, 5:24 AM

#

the actual data objects inside though arent thread safe

graceful slate Jul 22, 2022, 5:24 AM

#

oh

queen goblet Jul 22, 2022, 5:24 AM

#

just the map is

#

so referances to the map are safe

graceful slate Jul 22, 2022, 5:24 AM

#

i have barely any experience in multithreading in java so yeah

queen goblet Jul 22, 2022, 5:24 AM

#

but the actual values they represent are not

#

Ask conclure about multithreading

#

i dont wanna explain and fuck that part up for you

graceful slate Jul 22, 2022, 5:25 AM

#

alright

queen goblet Jul 22, 2022, 5:25 AM

#

i typically get one thing wrong when explaining it

#Data management pain