Best way to store massive amounts of player data locally?

[rotund cobalt]

Hey guys!

I'm writing a plugin for my server which needs to store a bunch of additional data for every player. (e.g. nickname, settings, owned lands, friends, blocked, etc..)

Currently I'm storing in this in a yml file which looks something like this:

uuid:
  data1: value
  data2: value
  data3: value
  etc...

This works fine, however I need to be able to store the data for 10k+ players, which makes the yml file extremely big. If I try to load the file configuration I get memory heap warnings in chat.

So my question is, what is the best way to store lots of data for thousands of players locally?

[hazy marsh]

First of all a proper database sounds like its needed

At least in your case

but how many nodes per player r there approximately?

All things considered, if there are too many the issue you're facing might have to be handled by having more than one server instance, this is how the biggest servers handle thousands of players, there is only so much memory after all.

[rocky sand]

mysql maybe

or some external database like mongodb

because its structured so you can store object more easily

[hazy marsh]

I mean yeah a database couldn't hurt but the issue here seems to be the concurrent amount of data loaded into memory.

[rotund cobalt]

by the time I'm finished there will be about 20 I think

[hazy marsh]

20*10,000

hmm

[rotund cobalt]

the thing is they don't all need to be loaded at once, but I can't load an individual player without loading the entire file

as far as I was able to figure out

[hazy marsh]

wait

is all the data contained in one file?

[rotund cobalt]

yeah

[hazy marsh]

oh

lol

well perhaps have a file per player

or as orbyfied mentioned, a database if you're willing to invest in that

[rotund cobalt]

that's how I had it before, but then I ran into a different issue...

If I need to grab every player which is vanished for example... I gotta loop through all those thousands of files, load them, check the vanished value, and return a result

[hazy marsh]

why every player?

[rotund cobalt]

Well if I'm on the server, and I want to see a list of all vanished players... I've got to check if every player is vanished...

how else am I supposed to get the list :P

[hazy marsh]

does that include offline players?

[rotund cobalt]

yeah that's the thing

[hazy marsh]

hmm I mean, one way would be to lazy load necessary players

[rotund cobalt]

What is "lazy load"?

[hazy marsh]

hmm

well you know hypixel for instance

[rotund cobalt]

yep

[hazy marsh]

wait let me log on and take a screenshot

so here

they have this "friends" thing in your profile menu

where you can scroll through all friends regardless if one is offline or online

however they lazy load all friends, which means, they only load once they are actually needed to be there

a friend located on page 35 won't be loaded unless I would go all the way to page 35

I thought, maybe you could lazy load players similarly to how it's done there

[rotund cobalt]

hmm I see what you're saying, but I'm afraid I still don't understand how that can help me get a list of vanished players for example...

since I'm only loading the players once someone requests the list

[hazy marsh]

yes

but do you need all the players, as soon as someone is requesting the list?

[rotund cobalt]

yeah, all the vanished players.

[hazy marsh]

realistically speaking, that list could be gigantic

[rotund cobalt]

Ooooooh I see what you're saying. No I thought of that, it only displays 20 items at once and you need to request the next page for another 20

[hazy marsh]

ye

[rotund cobalt]

my problem is that to get any amount of vanihsed players I need to loop through potentially 10k offline players and check if they're vanished

[hazy marsh]

you could have a seperate data file where you save the amount of vanished and unvanished players

[rotund cobalt]

hmmmmm

[hazy marsh]

so whenever someone becomes vanished, increment the variable, whenever someone becomes unvanished, decrement the variable

actually

there's no need to involve a data file there

just on enable go through all files

or well, in your case hopefully that ought to be fine

unless you're getting millions of files

[rotund cobalt]

Ah yeah that could work. What about for a more complicated situation...

Each player has a list of friends (stored as uuids) If I want to get all players who share a mutual friend, the way I'm doing that now is to loop through every offline player, then loop through all their friends, and check if if my friend list contains one of those items. As you can see this is becomes a disaster for large amounts of players

[hazy marsh]

so you wanna fetch mutual friends between a and b at any point of time

[rotund cobalt]

yeah something like that

atm it's just way too many for loops

[hazy marsh]

yeah, well in data structure theory that sounds like an advanced graph, but well one way would be to do this eagerly

if Abo adds Boo as a friend, then if Dodo adds Boo as a friend you add (Abo,Dodo) as a pair under the list of mutual friends for Boo

this requires discipline as when either Abo, or Dodo removes Boo, that pair has to vanish from Boo's "is-mutal-friend-to list"

so then let's say you do wanna check for mutual friends, now you'd just have to look at that one list

[rotund cobalt]

oh I see! Yeah I think I'l give that a try. Thanks for the help, and sorry for being so stupid about all this; a lot of this is new to me 😅

[hazy marsh]

lol no worries, I mean these are the issues even the biggest tech companies struggle with

actually not entirely sure if what I said will work entirely as planned but give it a shot

[sonic crystal]

staff channels kek 😂